Frederik Werner's Website http://www.frederikwerner.de/ Personal Website and Blog Sun, 23 Feb 2020 10:39:32 +0000 en-US hourly 1 https://wordpress.org/?v=5.3.2 Creating more Business Value with Product Driven Analytics https://www.frederikwerner.de/blog/analytics/creating-more-business-value-with-product-driven-analytics/ Sun, 23 Feb 2020 10:39:32 +0000 https://www.frederikwerner.de/?p=291 When working in Analytics, it’s easy to view all the requests you receive as disconnected, one-off events. This is the traditional approach for Analytics departments, where Requests would be collected, prioritized and answered accordingly. But there are some issues with this way of handling things: It’s hard to see the bigger picture behind those business […]

The post Creating more Business Value with Product Driven Analytics appeared first on https://www.frederikwerner.de/

]]>
When working in Analytics, it’s easy to view all the requests you receive as disconnected, one-off events. This is the traditional approach for Analytics departments, where Requests would be collected, prioritized and answered accordingly. But there are some issues with this way of handling things:

  • It’s hard to see the bigger picture behind those business questions, especially between departments. Different people may have the same questions but ask them in a different way, leading to different answers.
  • When requests are handled individually, it becomes hard to maintain a standard for the way they should be answered, especially if you are working as part of a team.
  • Since not all stakeholders are equally Data-savy, they may ask for the wrong thing without letting you know what the questions is they are trying to answer.

This leads to some awkward situations. People will ask for the wrong or suboptimal reports. You as the analyst will become more and more disconnected from the business and don’t know what actually matters to your stakeholders. In bigger teams, the same questions might be answered differently if they are asked by different people. Ultimately, you will run into a situation where people won’t trust your reports and hesitate to request them entirely. But what can you do?

How to build products

Let’s get some inspiration by looking at how companies approach product development. The classic approach roughly looks like this:

  1. Concept phase. Based on day-to-day observations, market research, trends, or customer pains, Product Management or Sales would come up with an idea on how to deliver value to the customer. Those ideas would be mapped with the abilities and skills of the company to get an idea for a new product.
  2. Market research phase. In this phase, research would be conducted to scout the potential of the product idea. Researchers would look at the size of the potential audience and possible gains.
  3. Business analysis phase. Now that we know what we want to do and what the audience is, Business Analysts would take that idea and see what costs and resources are required to develop a product. They would also estimate the sales volume and develop the business case.
  4. Development phase. Next, Product Development takes over to develop and test the actual product.
  5. Launch phase. Once we have our product, Marketing and Go-to-Market take over to bring the product to market and let the target audience know about it.
  6. Maintenance phase. Once customers are using the product, bugs need to be fixed and users get educated on how to use the product best.
  7. Sunset phase. When the product gets older over time, it will eventually be taken off the market and retired.

This circle would then repeat with the next product that is developed. But with the rise of agile digital development, things have changed quite a bit. Now companies focus on delivering initial value as fast as possible by building minimal-viable-products (MVPs), and then continuously develop them further in product iterations and deliver increments regularly.

Stop building dashboards, start building Data Products!

So, what can we learn from this? Instead of handling every request individually, we should start developing data products for our internal audience. If this means to either build a report or dashboard is up to us as the experts. To do this, we need to be vigilant and get to know our audience quite well. Who are the people you are working with? What are their tasks and pains? How do they deliver value to the business and how can you help them be successful? How experienced are they with data-driven-decisioning?

Once you know your audience, build a Data Product roadmap for your company and update it regularly. Think about the ways you will help your stakeholders be successful and prioritize your work by how big the audience is and how big the value your product delivers is. Then start building and let your business know what you are doing. For this you need to define the MVP scope to build something that is actually useful in a short amount of time. Discuss with your stakeholders what they need to find value in your product.

When that product is developed, you need to market it to your audience and let people know about it. For this you need to ask yourself how you will get the message to your target audience and what that message should be. Why should they trust you and invest their time in using your product?

But things don’t end here: Now that your product has some actual users, you need to collect and incorporate their feedback in product development. On top of the next steps on the product roadmap, there might be bugs you need to fix and UX issues you need to look into. Also, don’t forget that you need to document your product and educate your users on the best way to get value out of it.

At the end of the road, the time will come to finally sunset your product. Business changes, and so does the need for Data Products. You need to keep a close relation to your stakeholders, aka your target audience, and check in with them regularly on how to continue to deliver value.

Agile Analytics

Now we have defined what the ideal way to develop a modern Analytics practice is. The transition to such a way of working might be hard, but definitely worth the effort. It will lead to satisfied stakeholders and a clear understanding of how your team delivers value to the business.

A good way to start thinking in an agile manner is to ask yourself what the story behind your daily requests is today. What are your stakeholders trying to achieve with your report? Are they doing it in the best way possible? Is there some potential for a new Data Product? Who else would be interested in that?

If agile is completely new to you, it might help to educate yourself on methods like SCRUM and Kanban. A tool like Jira will help to keep an eye on your tasks, priorities and the backlog. Once this is clear to your team internally, give stakeholders access to your tool so they can see what the status of their product is (and why there may be other priorities right now). This will help you manage demand and show that you might need more resources to keep up with it.

The post Creating more Business Value with Product Driven Analytics appeared first on https://www.frederikwerner.de/

]]>
Using Adobe Launch 🚀 for configuration management https://www.frederikwerner.de/blog/tag-management/using-adobe-launch-for-configuration-management/ Sat, 15 Feb 2020 12:19:41 +0000 https://www.frederikwerner.de/?p=277 Adobe Launch (, by Adobe) ist the tag manager that comes for free with the Adobe Experience Cloud. You can use it to manage and orchestrate all the marketing tags on your websites (not only from Adobe) and reduce integration efforts for the mobile Adobe SDKs to just copy-pasting from the launch interface. For a […]

The post Using Adobe Launch 🚀 for configuration management appeared first on https://www.frederikwerner.de/

]]>
Adobe Launch (, by Adobe) ist the tag manager that comes for free with the Adobe Experience Cloud. You can use it to manage and orchestrate all the marketing tags on your websites (not only from Adobe) and reduce integration efforts for the mobile Adobe SDKs to just copy-pasting from the launch interface. For a tutorial, look below on this post.

Launch is the successor to the Adobe Dynamic Tag Manager (DTM) which was used in the past, formerly known as Satellite. In 2018 Adobe announced plans to retire DTM fully by 2021 (source: https://medium.com/launch-by-adobe/dtm-plans-for-a-sunset-3c6aab003a6f). So if you are still using DTM, you should start migrating to Launch, which is only a one-click-action under ideal circumstances.

Launch can host scripts for pre-tailored use cases (called Extensions) or custom code. With rules, we define when and where things should happen. Last, Data Elements let us define variables that we can use across all our tags to consolidate the tracked data points. But did you know that we can define static values and access them through the _satellite object? Let’s explore that!

Data Elements supercharged

When creating a new Data Element, Launch gives us a lot of options for the core extension. The obvious Element Types include web specific stuff like the Domain, Referrer or Query Parameters. There are some shorthand method for things like the Session Count, Time Spent or Visitor Retention. But for our example, we want to create two types of Elements: One Random Number and a Constant:

Data Element with Random Value
Data Element with Constant Value

With those two Data Elements, let’s publish our Library and take a look at our browser console. The _satellite object has a method getVar(“name”) that lets us get the value of a Data Element and takes the Data Element’s name as an argument. To get the constant value from the screenshot above, we would use _satellite.getVar(“test_constant”). Let’s try it out and get the value of both our random and constant Data Element:

There are two things to note here: The value returned for a Constant is always a string, even though we gave it an integer in Launch. If we wanted an integer, we would need to wrap it like parseInt(_satellite.getVar(“test_constant”)). Second, the random value changes every time we get it, respecting the boundaries we defined.

There are a lot of applications for this functionality that come to mind. For example, we could define a prefix for some tracking variables and change it without the need to ask a developer. At my company, we use sampling for our media tracking (to save on server calls). With Data Elements, we can have an evaluation like parseInt(_satellite.getVar(“test_constant”)) > _satellite.getVar(“test_random”) to decide if an interaction should be captured for tracking or not. If we ever want to change the sampling rate, we can do this completely in Launch without the need to touch any code!

There is only one downside to this: While it works like a charm on the web, we don’t have any equivalent for mobile Apps at time of writing. If you too would like to have this, please upvote my post on the Experience League.

Tutorial: Effortless Experience Cloud Implementation with Adobe Launch

The obvious use case for Launch is to manage your Experience Cloud Tags on your website. For a basic implementation of Adobe Analytics and Adobe Target you would first add the Extensions to Launch. Launch will automatically try to fetch as much configuration values as possible and prompt you to fill in whatever is still needed. In the end, it could look like this:

Next, you would add a Rule to fire when the Launch Library is loaded to load Target, fire its Page Load Request, and then send the Analytics Beacon:

Last step: Publish your Library and copy-paste the code from the Environments tab. This is what you would give to the developers of your website:

Done! Congratulations, you now have a basic but state-of-the-art Experience Cloud implementation of Launch, Target, and Analytics along the Experience Cloud ID Service. For validation, go to your website and enter “_satellite.setDebug(true)” in your browser console. This will output the launch debug lines on pageload. Refresh your page and look at the console to see what is going on:

Awesome! Launch is working and we can start testing our Website with Target and measure the effect with Analytics!

The post Using Adobe Launch 🚀 for configuration management appeared first on https://www.frederikwerner.de/

]]>
Analysis Workspace Hacks (AGE) – Metric Targets https://www.frederikwerner.de/blog/adobe-analytics/analysis-workspace-hacks-age-metric-targets/ Sat, 25 Jan 2020 16:15:09 +0000 https://www.frederikwerner.de/?p=251 This is a post in the Adam-Greco-Edition (AGE) series of posts. They aim to iterate on some great posts by Adam Greco, showing some different approaches to achieve similar things. In another great Post, Adam Greco showed how we can have Metric Targets in Analysis Workspace. His approach includes setting up a Data Source to […]

The post Analysis Workspace Hacks (AGE) – Metric Targets appeared first on https://www.frederikwerner.de/

]]>

This is a post in the Adam-Greco-Edition (AGE) series of posts. They aim to iterate on some great posts by Adam Greco, showing some different approaches to achieve similar things.

In another great Post, Adam Greco showed how we can have Metric Targets in Analysis Workspace. His approach includes setting up a Data Source to import Goals to a Custom Event.

This is a very nice approach, but has some serious limitations. Because it utilizes Data Sources, all their limitations apply (see documentation). Most importantly, data can not be deleted or changed once it has been imported. Also we need to sacrifice Custom Events for every Goal we set. The setup is also very involved and not suited for non-techie people.

What I would like to have is a Goal Metric that does not use valuable Custom Events, is changeable over time, and understandable and usable by non-technical users. As a plus, it would be nice if we could change it via both API and frontend. Maybe we can use a Calculated Metric for that?

Calculated Goal Metric

First, we need to tell Analytics if it should output our Goal in the Calculated Metric. To do this, we will be using the IF-function in the Calculated Metric Builder:

First the easy part: To tell Analytics what our Goal should be, put the Target value in the value_if_true box via a Static Number. Then, put a Static Number with “0” in the value_if_false box like this:

To keep it simple, we want to define a monthly Goal. The next step is to drop the “Month” Dimension in the logical_test box. This will open an overlay to select the month that we want to set for our Goal. This adds an Ad Hoc Segment for that specific Month, January 2019 in my case. In the Segment Container, drop any non-zero Metric like Occurences or Unique Visitors:

So, what this does is check if the current Month is the one we selected. If it is, it outputs our Goal, or 0 if not. Now, we just repeat this for all Months we want to cover and connect the functions with the Addition operator:

If i now use this new Metric for 2019 with the Month Dimension in Workspace, it gives me a nice Monthly Goal Metric and sums up all the values as Total:

Done! Whenever we want to change our Goals, we just have to edit our Metric. We could use it further to calculate the deviation from our Goal in Workspace. But there is still more…

Automate all the things

Adobe Analytics offers a huge set of APIs to change and automate almost everything. For our case, it would be nice to be able to update our Goals via an API. So let’s head to the Swagger UI to play around with that and authenticate on it!

First, we need the ID of our Calculated Metric. Open our newly created Goal Metric via the Components Menu in Analytics. The last Section of the URL includes the ID, starting with “cm”, like “components/calculatedMetrics/edit/cm2330_5e25a21g0e2a245s2147d09e“. In Swagger, extend the calculatedmetrics-section and the GET “/calculatedmetrics/{id}” function.

Click the Try-out button and enter the ID of your metric in the text box. In the expansion selector, make sure the “definition” part is selected. Once you click Execute, it gives you the request and response for that function. My response looks like below (without the IDs), where we can see all the Goals I defined (compare with the Screenshot above). If we wanted to change a Goal, we could update a part like “then”: 20000, to “then”: 22000, or whatever we like, and use the PUT “/calculatedmetrics/{id}” function to update the Goal. Amazing!

{
  "id": "cm2330_5e25a21g0e2a245s2147d09e",
  "name": "Goal Test",
  "description": "",
  "rsid": "id",
  "owner": {
    "id": id
  },
  "polarity": "positive",
  "precision": 0,
  "type": "decimal",
  "definition": {
    "formula": {
      "func": "add",
      "col1": {
        "func": "add",
        "col1": {
          "func": "add",
          "col1": {
            "func": "add",
            "col1": {
              "func": "add",
              "col1": {
                "func": "add",
                "col1": {
                  "func": "add",
                  "col1": {
                    "func": "add",
                    "col1": {
                      "func": "add",
                      "col1": {
                        "func": "add",
                        "col1": {
                          "func": "add",
                          "col1": {
                            "func": "add",
                            "col1": {
                              "func": "if",
                              "description": "If",
                              "cond": {
                                "func": "calc-metric",
                                "formula": {
                                  "func": "metric",
                                  "name": "metrics/visitors",
                                  "description": "Unique Visitors"
                                },
                                "version": [
                                  1,
                                  0,
                                  0
                                ],
                                "filters": [
                                  {
                                    "func": "segment-ref",
                                    "description": "Month = Jan 2019 (Jan 01-Jan 31)",
                                    "id": "id"
                                  }
                                ]
                              },
                              "then": 20000,
                              "else": 0
                            },
                            "col2": {
                              "func": "if",
                              "description": "If",
                              "cond": {
                                "func": "calc-metric",
                                "formula": {
                                  "func": "metric",
                                  "name": "metrics/visitors",
                                  "description": "Unique Visitors"
                                },
                                "version": [
                                  1,
                                  0,
                                  0
                                ],
                                "filters": [
                                  {
                                    "func": "segment-ref",
                                    "description": "Month = Feb 2019 (Feb 01-Feb 28)",
                                    "id": "id"
                                  }
                                ]
                              },
                              "then": 20001,
                              "else": 0
                            }
                          },
                          "col2": {
                            "func": "if",
                            "description": "If",
                            "cond": {
                              "func": "calc-metric",
                              "formula": {
                                "func": "metric",
                                "name": "metrics/visitors",
                                "description": "Unique Visitors"
                              },
                              "version": [
                                1,
                                0,
                                0
                              ],
                              "filters": [
                                {
                                  "func": "segment-ref",
                                  "description": "Month = Mar 2019 (Mar 01-Mar 31)",
                                  "id": "id"
                                }
                              ]
                            },
                            "then": 20002,
                            "else": 0
                          }
                        },
                        "col2": {
                          "func": "if",
                          "description": "If",
                          "cond": {
                            "func": "calc-metric",
                            "formula": {
                              "func": "metric",
                              "name": "metrics/visitors",
                              "description": "Unique Visitors"
                            },
                            "version": [
                              1,
                              0,
                              0
                            ],
                            "filters": [
                              {
                                "func": "segment-ref",
                                "description": "Month = Apr 2019 (Apr 01-Apr 30)",
                                "id": "id"
                              }
                            ]
                          },
                          "then": 20003,
                          "else": 0
                        }
                      },
                      "col2": {
                        "func": "if",
                        "description": "If",
                        "cond": {
                          "func": "calc-metric",
                          "formula": {
                            "func": "metric",
                            "name": "metrics/visitors",
                            "description": "Unique Visitors"
                          },
                          "version": [
                            1,
                            0,
                            0
                          ],
                          "filters": [
                            {
                              "func": "segment-ref",
                              "description": "Month = May 2019 (May 01-May 31)",
                              "id": "id"
                            }
                          ]
                        },
                        "then": 20004,
                        "else": 0
                      }
                    },
                    "col2": {
                      "func": "if",
                      "description": "If",
                      "cond": {
                        "func": "calc-metric",
                        "formula": {
                          "func": "metric",
                          "name": "metrics/visitors",
                          "description": "Unique Visitors"
                        },
                        "version": [
                          1,
                          0,
                          0
                        ],
                        "filters": [
                          {
                            "func": "segment-ref",
                            "description": "Month = Jun 2019 (Jun 01-Jun 30)",
                            "id": "id"
                          }
                        ]
                      },
                      "then": 20005,
                      "else": 0
                    }
                  },
                  "col2": {
                    "func": "if",
                    "description": "If",
                    "cond": {
                      "func": "calc-metric",
                      "formula": {
                        "func": "metric",
                        "name": "metrics/visitors",
                        "description": "Unique Visitors"
                      },
                      "version": [
                        1,
                        0,
                        0
                      ],
                      "filters": [
                        {
                          "func": "segment-ref",
                          "description": "Month = Jul 2019 (Jul 01-Jul 31)",
                          "id": "id"
                        }
                      ]
                    },
                    "then": 20006,
                    "else": 0
                  }
                },
                "col2": {
                  "func": "if",
                  "description": "If",
                  "cond": {
                    "func": "calc-metric",
                    "formula": {
                      "func": "metric",
                      "name": "metrics/visitors",
                      "description": "Unique Visitors"
                    },
                    "version": [
                      1,
                      0,
                      0
                    ],
                    "filters": [
                      {
                        "func": "segment-ref",
                        "description": "Month = Aug 2019 (Aug 01-Aug 31)",
                        "id": "id"
                      }
                    ]
                  },
                  "then": 20007,
                  "else": 0
                }
              },
              "col2": {
                "func": "if",
                "description": "If",
                "cond": {
                  "func": "calc-metric",
                  "formula": {
                    "func": "metric",
                    "name": "metrics/visitors",
                    "description": "Unique Visitors"
                  },
                  "version": [
                    1,
                    0,
                    0
                  ],
                  "filters": [
                    {
                      "func": "segment-ref",
                      "description": "Month = Sep 2019 (Sep 01-Sep 30)",
                      "id": "id"
                    }
                  ]
                },
                "then": 20008,
                "else": 0
              }
            },
            "col2": {
              "func": "if",
              "description": "If",
              "cond": {
                "func": "calc-metric",
                "formula": {
                  "func": "metric",
                  "name": "metrics/visitors",
                  "description": "Unique Visitors"
                },
                "version": [
                  1,
                  0,
                  0
                ],
                "filters": [
                  {
                    "func": "segment-ref",
                    "description": "Month = Oct 2019 (Oct 01-Oct 31)",
                    "id": "id"
                  }
                ]
              },
              "then": 20009,
              "else": 0
            }
          },
          "col2": {
            "func": "if",
            "description": "If",
            "cond": {
              "func": "calc-metric",
              "formula": {
                "func": "metric",
                "name": "metrics/visitors",
                "description": "Unique Visitors"
              },
              "version": [
                1,
                0,
                0
              ],
              "filters": [
                {
                  "func": "segment-ref",
                  "description": "Month = Nov 2019 (Nov 01-Nov 30)",
                  "id": "id"
                }
              ]
            },
            "then": 20010,
            "else": 0
          }
        },
        "col2": {
          "func": "if",
          "description": "If",
          "cond": {
            "func": "calc-metric",
            "formula": {
              "func": "metric",
              "name": "metrics/visitors",
              "description": "Unique Visitors"
            },
            "version": [
              1,
              0,
              0
            ],
            "filters": [
              {
                "func": "segment-ref",
                "description": "Month = Dec 2019 (Dec 01-Dec 31)",
                "id": "id"
              }
            ]
          },
          "then": 20011,
          "else": 0
        }
      },
      "col2": {
        "func": "if",
        "description": "If",
        "cond": {
          "func": "calc-metric",
          "formula": {
            "func": "metric",
            "name": "metrics/visitors",
            "description": "Unique Visitors"
          },
          "version": [
            1,
            0,
            0
          ],
          "filters": [
            {
              "func": "segment-ref",
              "description": "Month = Jan 2020 (Jan 01-Jan 31)",
              "id": "id"
            }
          ]
        },
        "then": 20012,
        "else": 0
      }
    },
    "func": "calc-metric",
    "version": [
      1,
      0,
      0
    ]
  }
}

Steps to create Goal Metric in Analysis Workspace:

  1. Create new Calculated Goal Metric

    Open the Calculated Metric Builder and drop as many IF functions as you would like for your Goal.

  2. Define Month Dimensions for each Function

    Select the Month that we want our Goal to be valid for.

  3. Set Goals in true-section and 0 in false-section

    In the value_if_true-section, we need to define our Goal and set the other Container to 0.

  4. Use Metric in Workspace

    Drop the Metric in a Freeform Table with the Month Dimension to see our Monthly Goals.

  5. Use API to update Metric

    If needed, use the Analytics API to update the Goals you defined before.

The post Analysis Workspace Hacks (AGE) – Metric Targets appeared first on https://www.frederikwerner.de/

]]>
Analysis Workspace Hacks (AGE) – Average Daily Unique Visitors https://www.frederikwerner.de/blog/adobe-analytics/analysis-workspace-hacks-age-average-daily-unique-visitors/ Sat, 25 Jan 2020 14:46:26 +0000 https://www.frederikwerner.de/?p=241 This is a post in the Adam-Greco-Edition (AGE) series of posts. They aim to iterate on some great posts by Adam Greco, showing some different approaches to achieve similar things. In one of his posts Adam Greco shows a way to replicate the Daily Unique Visitors Metric from Reports & Analytics in Analysis Workspace. His […]

The post Analysis Workspace Hacks (AGE) – Average Daily Unique Visitors appeared first on https://www.frederikwerner.de/

]]>

This is a post in the Adam-Greco-Edition (AGE) series of posts. They aim to iterate on some great posts by Adam Greco, showing some different approaches to achieve similar things.

In one of his posts Adam Greco shows a way to replicate the Daily Unique Visitors Metric from Reports & Analytics in Analysis Workspace. His approach involves creating a Calculated Metric for a given time range, summing up the Visitors for each day.

There are some limitations to that approach. The obvious one is that we need a new metric for each date range we want to analyze; We can’t use a 7-day Metric if there are 8 days to analyze. Second, Visitors are not deduplicated but summed up over all days in the reporting window (just as in the old interface); So a Visitor visiting our site three times would be counted as three Visitors. Last, the name could mislead our users into thinking they are looking at an average daily amount of Visitors instead of the sum, which I never liked in the past.

The last issue is solved simply by dividing our sum by the amount of days in the Metric. For the second issue, we would use one 7-day segment instead of seven 1-day segments, effectively deduplicating our Visitors. But what can we do to avoid having to build a separate Metric for every time frame?

Introducing: Approximate Count Distinct

Luckily, Analytics has learned to count things with the Approximate Count Distinct function in the Calculated Metric Builder. With this function, we are able to count how many different Dimension values we have in a given Dimension during the Reporting Window. Once we drop it into the Definition, it presents itself like this:

As the Dimension, we will choose “Day” for our little use case, so drop it in there like this:

We name our Metric “Days” and save it. Let’s see what it does and throw it into a Freeform Table:

Yay, it works! Now we have a Metric returning the number of days in a given Reporting Window. Now we just need to divide our Unique Visitors by those counted days:

Now let’s see it side by side with Visitors and our Days Metric:

Awesome! Now we have exactly what we wanted! For Dec 2019, our 202 Visitors are divided by all the 30 days in December. Wait, what?

Counting things is hard

This is where the “approximate” part of our function name comes in. For a system like Adobe Analytics, it’s a tremendous task to count how many different values for a Dimension are in a flexible Reporting Window. According to the documentation, they aim for 95% accuracy 95% of the time.

This is a good precision given that Analytics has to handle Terabytes of data. Things become more apparent once we include decimals to our days Metric:

We can see that there are approximately 30.22 days in December, which is quite precise. While it appears like a simple task for us (since people know how many days are there in any month), the system is actually doing a good job approximating it.

All done!

Now we have our Average Daily Unique Visitors exactly how we want them. By using our counting function, we can change the Reporting Window without having to switch the Metric. We can do the same by using Weeks, Months, or Years as a Dimension like in the old interface. On top of that, we are also able to use the “Hour” Dimension!

I generally recommend exploring the possible use cases for Approximate Count Distinct on your site. The documentation already shows how to count the active Customers, but there is much more to explore. How many pages of your site are visited over time? How do they differ with certain entry pages? How many different search requests are issued on your search? How many videos played? There is so much more to explore!

Steps to create Average Daily Unique Visitors in Analysis Workspace:

  1. Create “Days” Metric

    Open the Calculated Metric Builder and drop the Approximate Count Distinct Function in there. Use “Day” as Dimension. Save the Metric and check if it looks like in the example.

  2. Check if Metric works as expected

    Drop the Metric alongside your Unique Visitors in a Freeform Table.

  3. Create “Average Daily Unique Visitors” Metric

    Open the Metric Builder and divide Unique Visitors by the created “Days” Metric.

  4. Put it all together

    Drop both new Metrics in Analysis Workspace with any Dimension you like for any Reporting Window.

  5. Go beyond

    Create more Calculated Metrics like this using the Year, Month, Week, or Hour Dimensions.

The post Analysis Workspace Hacks (AGE) – Average Daily Unique Visitors appeared first on https://www.frederikwerner.de/

]]>
Analysis Workspace Hacks – Next and Previous Page Report https://www.frederikwerner.de/blog/adobe-analytics/analysis-workspace-hacks-next-and-previous-page-report/ Sat, 18 Jan 2020 10:50:23 +0000 https://www.frederikwerner.de/?p=226 Analysis Workspace is the most capable solution for Web Analysts today. It allows us to switch between building a Dashboard or old-school Report or something in the middle on the fly. It has surpassed the old Reports & Analytics Interface in functionality and workflow effectiveness and leaves you longing for it once you start using […]

The post Analysis Workspace Hacks – Next and Previous Page Report appeared first on https://www.frederikwerner.de/

]]>
Analysis Workspace is the most capable solution for Web Analysts today. It allows us to switch between building a Dashboard or old-school Report or something in the middle on the fly. It has surpassed the old Reports & Analytics Interface in functionality and workflow effectiveness and leaves you longing for it once you start using different solutions.

But there is one thing that is not that awesome in Analysis Workspace yet: Pathing. Once you activate Pathing for a custom prop, the old interface gives you Next and Previous Reports for that prop, just like with the Page Dimension:

Pathing in Reports & Analytics

As a result we get a nice table with the Next or Previous Dimension Items for a given Item.

Hacking Analysis Workspace’s Flow Visualizations

The closest thing to that functionality is the Flow Visualization in Analysis Workspace. It allows us to see a Flow of Users between Dimension Items or even across different Dimensions. It is way more flexible than the old interface, since we don’t even need to enable Pathing for props that we want to analyze:

But those Graphs can get really large. And since Workspace is creating those Graphs, it needs to have the information somewhere, right? So, let’s look at that! If we right click on a node in the Graph, we can trend that Connection:

As result, we get a Graph showing us the traffic across that connection:

On that Graph, let’s display the Freeform Table (that is secretly behind every Vis in Workspace):

On the resulting Table, we have a trended view of that Graph. But now we can see the Segment that is responsible for that Graph:

Let’s hack that Segment! In my Case, it looks like this. But what would happen, if we change the third Item to not only include a specific page, but any page? Let’s try that! Change the specific Page to a simple “exists” condition:

Once saved, we now have a few more Occurences for our Table. Since we want to look at the next Page, replace the Time Dimension with the Page Dimension. And there we have it! Now we have a Freeform Table showing us the next Page for our selected Page:

Next Page report in Analysis Workspace

Now we can look at all the next Pages for a given Page within Analysis Workspace. Awesome! But I know: This is a very involved way to achieve this. We can’t really demand this from our non-technical users. If you want an easier way, vote for my idea on the Adobe Analytics Forums!

The post Analysis Workspace Hacks – Next and Previous Page Report appeared first on https://www.frederikwerner.de/

]]>
Analysis Workspace Hacks – Link Events on Page Reports https://www.frederikwerner.de/blog/adobe-analytics/analysis-workspace-hacks-link-events-on-page-reports/ Sat, 18 Jan 2020 10:13:49 +0000 https://www.frederikwerner.de/?p=220 Adobe Analytics gives us two types of events to use for our tracking implementation. With Page Tracking (calling s.t() in Websites or trackState() in Apps) we are supposed to measure when a page has been viewed. If we want to measure interactions on a given page, we would use Custom Link Tracking (s.tl() in Web […]

The post Analysis Workspace Hacks – Link Events on Page Reports appeared first on https://www.frederikwerner.de/

]]>
Adobe Analytics gives us two types of events to use for our tracking implementation. With Page Tracking (calling s.t() in Websites or trackState() in Apps) we are supposed to measure when a page has been viewed. If we want to measure interactions on a given page, we would use Custom Link Tracking (s.tl() in Web and trackAction() in Apps) for that.

The reasoning behind that is quite simple. If there was only one function, we would either end up with increased Page Views for every on-page event or have to take care of the distinction ourself by using valuable props or eVars. So from a simplicity standpoint this approach makes a lot of sense.

But there is one problem: When using Custom Link Tracking, you can not set a pageName for that call. Adobe Analytics just ignores whatever you set for the pageName, because pageNames only make sense in the context of Page Tracking. This leaves us in an awkward position if we want to know on which page a Link event happend: Either we have to use a prop and set the pageName with every Page or Custom Link event or an eVar with Visit Expiration on every Page Tracking. Yikes.

As an example, let’s say we have some videos on our website that are tracked by Custom Link Events. It would be interesting to know how many videos are started on a given page, right? But whops, if we use our Video event on the Pages Report, we don’t see any pages there (because they are thrown away on a Custom Link Call):

No Custom Link Events on our Pages Report

Attribution IQ to the rescue!

This would be a very depressing post if that was the end of the story. Quite a while ago, Adobe invented Attribution IQ. This awesome feature allows us to change how Analysis Workspace attributes Events on the fly. There are some differences between the models available depending on your license, but the basics we need here are universally available.

When you drop a Metric to a Freeform Table, try clicking the little gear icon on the metric. In the settings, we have the Attribution settings at the bottom:

This brings up the Attribution Model Overlay, letting us choose how we want to attribute the Events to our Dimensions. For our little use case, we want to know what the last Page in a Visit was before the Video was started. For this, we choose the Last Touch Model in a Visit Window:

This gives us a little comment below the Metric name, telling us about the different Attribution setting. Comparing our Metric with this setting to the original Metric, we can now see on which Page our Event happened:

This opens up endless possibilities and frees us from the need to write our Page Name into a lot of different props and eVars with every Page or Custom Link Event. Awesome!

The post Analysis Workspace Hacks – Link Events on Page Reports appeared first on https://www.frederikwerner.de/

]]>
Migrating from Android’s BroadcastReceiver to Google Play Install Referrer API with Adobe Analytics https://www.frederikwerner.de/blog/adobe-analytics/migrating-from-androids-broadcastreceiver-to-google-play-install-referrer-api-with-adobe-analytics/ Wed, 15 Jan 2020 18:12:13 +0000 https://www.frederikwerner.de/?p=201 Adobe Analytics can track not only websites, but mobile apps just the same. This is achieved by using the Adobe Experience Platform Mobile SKDs for native iOS and Android apps. One very interesting part of tracking mobile apps is known as acquisition tracking, which looks at how users found the tracked app. To help with […]

The post Migrating from Android’s BroadcastReceiver to Google Play Install Referrer API with Adobe Analytics appeared first on https://www.frederikwerner.de/

]]>
Adobe Analytics can track not only websites, but mobile apps just the same. This is achieved by using the Adobe Experience Platform Mobile SKDs for native iOS and Android apps.

One very interesting part of tracking mobile apps is known as acquisition tracking, which looks at how users found the tracked app. To help with this, Adobe exposes some functionality in their SDKs to listen for the events that the mobile operating system is using to tell apps about the way they have been installed. This happens “automagically” on iOS but needs some custom implementation on Android.

BroadcastReciever and Install Referrer API on Android

Adobe requires to use a very old implementation method called BroadcastReceiver. That method relies on the Google Play Store App sending a message (a broadcast) to the app that has just been installed, telling it about the details of the install (like which marketing campaign has been used). This is a very old method on Android and has a successor called Install Referrer API since 2017. Because of this Google announced to retire the old method by March 1, 2020 back in November 2019 and started sending out reminder emails.

By this date the Google Play Store App will stop sending out the broadcasts the Adobe SKDs rely on. This will only happen in newer versions of the Play Store App. As long as the user does not update that app, broadcasts will still be sent. If the old versions of the Play Store remain functional users could still install apps and the Play Store will send the broadcasts.

That portion of users will decrease with updates over time but leaves marketeers with a scenario where both old and new methods will be used. However, the good message is that there is no harm in still having the old code in your app, since it will only listen for a broadcast it will never receive.

As a migration strategy, app maintainers could implement both methods. But this is not a good idea, since it would leave us with two events for the same install. Because the new method has been around since 2017, we can expect a high adoption by now. Therefore, we should only focus on the state-of-the-art method.

New Adobe methods for Install Referrer

Adobe announced in the Experience Cloud Release Notes that a new version of the Mobile Services and Mobile SDKs will be released on January 16, 2020. The notes say:

  • Acquisition – Added a new API, Analytics.processGooglePlayInstallReferrerUrl(final String url) , to support Google Play Install Referrer APIs.

At time of writing, those new versions are not available for download via Mobile Services or GitHub. There is no documentation as well, so we need to make an educated guess at what we need to do.

The parameter of that new API gives us a hint. “final String url” looks like the string we would get when using the getInstallReferrer() method with the Install Referrer API. Given that information, let’s implement the new method with our Adobe SDKs!

Disclaimer

The Adobe-official way of implementation might differ in the future. Also, those changes should not be made without the new SDK version available, since the old version does not have the API we need. This post was written on January 15, 2020, and things might have changed already.

Implementing Play Install Referrer with Mobile Services

The general method of implementing the new API is described in detail in the Android Developer Documentation. It involves connecting to Google Play to get the desired information and handling the response.

But there is one caveat: The information about the install method remains available for 90 days after the installation. Because of this, we need to implement a different broadcast receiver (there it is again!) to only send the referrer on the first app launch. Since I am no Android developer, I will rely on Martin Zeitler’s post on stackoverflow about how to implement this in conjunction with the old documentation.

To use the new Play API, we need to add the dependency to our build.gradle:

implementation 'com.android.installreferrer:installreferrer:1.1'

The next step is similar to the old broadcast receiver. In our AndroidManifest.xml, we replace the old intent receiver

... action android:name="com.android.vending.INSTALL_REFERRER" ...

with

... action android:name="android.intent.action.PACKAGE_FIRST_LAUNCH" ...

to get a notification about the first launch of our app.

Next, in our Java-file, we listen for that notification and pass it to the new Mobile Services API (might have missed some imports):

import com.adobe.mobile.*;
import android.content.BroadcastReceiver;
import com.android.installreferrer.api.InstallReferrerClient;
import com.android.installreferrer.api.InstallReferrerStateListener;
import com.android.installreferrer.api.ReferrerDetails;
import android.content.Context;
import android.content.Intent;

public class PackageStatusReceiver extends BroadcastReceiver implements InstallReferrerStateListener {

    protected static final String LOG_TAG = PackageStatusReceiver.class.getSimpleName();

    private InstallReferrerClient referrerClient;

    @Override
    public void onReceive(Context context, Intent intent) {
        if(intent.getAction() != null) {
            if(intent.getAction().equals(Intent.ACTION_PACKAGE_FIRST_LAUNCH)) {
                this.referrerClient = InstallReferrerClient.newBuilder(context).build();
                this.referrerClient.startConnection(this);
            }
        }
    }

    @Override
    public void onInstallReferrerSetupFinished(int responseCode) {
        switch (responseCode) {
            case InstallReferrerClient.InstallReferrerResponse.OK:
                Log.d(LOG_TAG, "InstallReferrer Response.OK");
                try {
                    ReferrerDetails response = referrerClient.getInstallReferrer();
                    String referrer = response.getInstallReferrer();
                    com.adobe.mobile.Analytics.processGooglePlayInstallReferrerUrl(referrer);
                    Log.d(LOG_TAG, "InstallReferrer " + referrer);
                    referrerClient.endConnection();
                } catch (RemoteException e) {
                    Log.e(LOG_TAG, "" + e.getMessage());
                }
                break;
            case InstallReferrerClient.InstallReferrerResponse.FEATURE_NOT_SUPPORTED:
                Log.w(LOG_TAG, "InstallReferrer Response.FEATURE_NOT_SUPPORTED");
                break;
            case InstallReferrerClient.InstallReferrerResponse.SERVICE_UNAVAILABLE:
                Log.w(LOG_TAG, "InstallReferrer Response.SERVICE_UNAVAILABLE");
                break;
            case InstallReferrerClient.InstallReferrerResponse.SERVICE_DISCONNECTED:
                Log.w(LOG_TAG, "InstallReferrer Response.SERVICE_DISCONNECTED");
                break;
            case InstallReferrerClient.InstallReferrerResponse.DEVELOPER_ERROR:
                Log.w(LOG_TAG, "InstallReferrer Response.DEVELOPER_ERROR");
                break;
        }
    }

    @Override
    public void onInstallReferrerServiceDisconnected() {
        Log.w(LOG_TAG, "InstallReferrer onInstallReferrerServiceDisconnected()");
    }
}

Now, Mobile Services should receive the data, since we called com.adobe.mobile.Analytics.processGooglePlayInstallReferrerUrl(referrer) with our referrer-string.

The post Migrating from Android’s BroadcastReceiver to Google Play Install Referrer API with Adobe Analytics appeared first on https://www.frederikwerner.de/

]]>
Building your own Web Analytics from Log Files – Part 6: Conclusion https://www.frederikwerner.de/blog/web-analytics/building-your-own-web-analytics-from-log-files-part-6-conclusion/ Fri, 10 Jan 2020 15:11:00 +0000 https://www.frederikwerner.de/?p=134 This is the sixth part of the six-part-series “Building your own Web Analytics from Log Files”. In this series we built a rather sophisticated logging and tracking functionality for our website. We used OpenResty to identify and fingerprint our users via cookies, stored that information to log files which were shipped to Elasticsearch and visualized […]

The post Building your own Web Analytics from Log Files – Part 6: Conclusion appeared first on https://www.frederikwerner.de/

]]>
This is the sixth part of the six-part-series “Building your own Web Analytics from Log Files”.

In this series we built a rather sophisticated logging and tracking functionality for our website. We used OpenResty to identify and fingerprint our users via cookies, stored that information to log files which were shipped to Elasticsearch and visualized with Kibana.

Web Analytics democratized

By using those techniques, we are able to use what we already have (log file processing) to answer questions about our users. Under best conditions this doesn’t even lead to a bigger technical footprint.

This way we can have deep insights into our user behavior without external tools. Even as a startup or hobby developer you are now able to put the user first on your digital platforms.

Next steps

While this series is done for now we have a starting point to further build our platform. With some frontend code we would be able to basically build something like Google Analytics by our self. If we are willing to put in some extra effort even more complex solutions like Adobe Analytics could be rebuilt.

Even if this is not necessary, with this project we have a good starting point for further extension. We can build basically any tool for user tracking with this. It was fun to build and write those articles, hope it was some fun for you too! See you next time!

The post Building your own Web Analytics from Log Files – Part 6: Conclusion appeared first on https://www.frederikwerner.de/

]]>
Building your own Web Analytics from Log Files – Part 5: Building our first Dashboard https://www.frederikwerner.de/blog/web-analytics/building-your-own-web-analytics-from-log-files-part-5-building-our-first-dashboard/ Thu, 09 Jan 2020 16:48:00 +0000 https://www.frederikwerner.de/?p=130 This is the fifth part of the six-part-series “Building your own Web Analytics from Log Files”. At this part of the series we have our log files in Elasticsearch with indices like “custom-filebeat-tracking-logs-7.4.0-2020.01.03”. First thing is to set up a Kibana index pattern for this. Kibana Configuration In Kibana we go to Management -> Index […]

The post Building your own Web Analytics from Log Files – Part 5: Building our first Dashboard appeared first on https://www.frederikwerner.de/

]]>
This is the fifth part of the six-part-series “Building your own Web Analytics from Log Files”.

At this part of the series we have our log files in Elasticsearch with indices like
“custom-filebeat-tracking-logs-7.4.0-2020.01.03”. First thing is to set up a Kibana index pattern for this.

Kibana Configuration

In Kibana we go to Management -> Index Patterns -> Create index pattern. As Index pattern we use “custom-filebeat-tracking-logs-*”, which gives us all the indices with our daily index pattern.

In the next step, we set the Time Filter field name to “@timestamp”. This is the timestamp that marks the point where Filebeat indexed the document. This is fine for now, we click “Create index pattern” and are done with this part!

Checking our Data

Now, let’s head to the Discover section in Kibana and look at our index pattern. And there it is: Our log entries show up like we wanted:

This is great already! We can look at the documents and see our fields just like we want them:

Building our Dashboard

With all that great data we can finally build a dashboard in Kibana. To get started, head over to the Visualize section and create a new Vertical Bar chart for our index pattern.

First, lets look at how many users we have on our site. Put the timestamp on the X-axis. On the Y-axis, select the Unique Count Aggregation for the json.visitor_id field. This gives us the count of visitor IDs, which is equivalent to the number of users on our site for a given time:

Let’s save that visualization. Next, we want to know how many new and returning users we have. We can use the previous visualization and modify it a bit. To do this, we split the series by the Terms in the json.new_visitor field. In the search bar, we need to filter for the entry requests, since we only then know about the user status:

Now we see how many of our users are new or returning! Let’s save that as well and create one more visualization for our first dashboard. We would like to know which entry page is popular for our new users. To do this, we create a new Data Table for our index pattern. Now we need to filter by user status and session entry, so we put “json.new_visitor : true and json.session_entry : true” in the search bar. Next we create Buckets for the json.uri field, which gives us the filename of the request:

Now we can see which pages are popular for our new users. With those three visualizations we can now build our first dashboard!

Head over to the Dashboard section in Kibana and create a new dashboard. On the top menu, click “Add” and add our three visualizations. Arrange them by dragging and dropping and resizing them. This gives us something like this, our first dashboard!

Next steps

Now it is up to you to further extend that dashboard. Why not do the same thing we did to get our users with our session ID to get the number of session? Or build a visualization to look at the top pages for all users? Or include referrers or URL parameters to know where your users are coming from and via which marketing campaigns? Or which browsers are most popular? Or specifically in- or exclude Google’s Bot from our dashboard?

The possibilities are endless. In the next part of this series we will step back and take a look at what we accomplished.

The post Building your own Web Analytics from Log Files – Part 5: Building our first Dashboard appeared first on https://www.frederikwerner.de/

]]>
Building your own Web Analytics from Log Files – Part 4: Data Collection and Processing https://www.frederikwerner.de/blog/web-analytics/building-your-own-web-analytics-from-log-files-part-4-data-collection-and-processing/ Wed, 08 Jan 2020 16:47:43 +0000 https://www.frederikwerner.de/?p=128 This is the fourth part of the six-part-series “Building your own Web Analytics from Log Files”. Legal Disclaimer: This post describes how to identify and track the users on your website using cookies, IP adresses and browser fingerprinting. The information and process described here may be subject to data privacy regulations under your legislation. It […]

The post Building your own Web Analytics from Log Files – Part 4: Data Collection and Processing appeared first on https://www.frederikwerner.de/

]]>
This is the fourth part of the six-part-series “Building your own Web Analytics from Log Files”.

Legal Disclaimer: This post describes how to identify and track the users on your website using cookies, IP adresses and browser fingerprinting. The information and process described here may be subject to data privacy regulations under your legislation. It is your responsibility to comply with all regulations. Please educate yourself if things like GDPR apply to your use case (which is very likely), and act responsibly.

In the last part we have built a configuration for OpenResty to generate user and session IDs and store them in browser cookies. Now we need a way to actually log and collect those IDs together with the requests our web server handles.

OpenResty Configuration

To be able to log our custom variables we need to announce them to Nginx. This is done right in the server-part of the configuration. We need variables for the user and session IDs, if the user or session is new and for the query parameters:

server {
    listen       8888;
    server_name  localhost;

    set $a_visitor_id '';
    set $a_visitor_new '';
    set $a_session_id '';
    set $a_session_entry '';
    set $a_query_args '';

In our header_filter_by_lua_block block we can now assign the values from the previous part to those variables:

ngx.var.a_visitor_id = a_visitor_id
ngx.var.a_visitor_new = a_visitor_new
ngx.var.a_session_id = a_session_id
ngx.var.a_session_entry = a_session_entry
ngx.var.a_query_args = a_query_args

Now Nginx knows about the variables and we are able to use them in our logging configuration. This is done in the http-part of the config. We define a new format called “tracking” and set an access_log for it. By doing this, we have a log line for every request to the web server in the format we specify. To make our life easy, we put our logs in a JSON format. We include a lot of useful variables we would log anyway plus our tracking variables:

log_format  tracking escape=none '{"remote_addr":"$remote_addr","remote_user":"$remote_user","scheme":"$scheme","uri":"$uri","connection_request":"$connection_requests","connection":"$connection","msec":"$msec","pid":"$pid","host":"$host","server_name":"$server_name","time":"$time_iso8601","request":"$request","status":"$status","size":"$body_bytes_sent","query_string":"$query_string","referer":"$http_referer","user_agent":"$http_user_agent","proxy_ip":"$http_x_forwarded_for","visitor_id":"$a_visitor_id","new_visitor":$a_visitor_new,"session_id":"$a_session_id","session_entry":$a_session_entry,"query_array":$a_query_args}';

access_log  logs/access-tracking.log  tracking;

At the end, we have an Nginx configuration with all the parts we need:

http {

    log_format  tracking escape=none '{"remote_addr":"$remote_addr","remote_user":"$remote_user","scheme":"$scheme","uri":"$uri","connection_request":"$connection_requests","connection":"$connection","msec":"$msec","pid":"$pid","host":"$host","server_name":"$server_name","time":"$time_iso8601","request":"$request","status":"$status","size":"$body_bytes_sent","query_string":"$query_string","referer":"$http_referer","user_agent":"$http_user_agent","proxy_ip":"$http_x_forwarded_for","visitor_id":"$a_visitor_id","new_visitor":$a_visitor_new,"session_id":"$a_session_id","session_entry":$a_session_entry,"query_array":$a_query_args}';

    access_log  logs/access-tracking.log  tracking;

    server {
        listen       8888;
        server_name  localhost;

        set $a_visitor_id '';
        set $a_visitor_new '';
        set $a_session_id '';
        set $a_session_entry '';
        set $a_query_args '';

        header_filter_by_lua_block {
            a_visitor_id = ngx.var["cookie_a_vid"]
            a_visitor_new = "false"
            if not a_visitor_id then
                a_visitor_id = ngx.md5(ngx.var.remote_addr..ngx.var.http_user_agent..(ngx.req.get_headers()["Accept"] or '')..(ngx.req.get_headers()["Accept-Encoding"] or '')..(ngx.req.get_headers()["Accept-Language"] or ''))
                a_visitor_new = "true"
            end

            a_session_id = ngx.var["cookie_a_sid"]
            a_session_entry = "false"
            if not a_session_id then
                a_session_id = ngx.md5(ngx.var.remote_addr..ngx.var.http_user_agent..(ngx.req.get_headers()["Accept"] or '')..(ngx.req.get_headers()["Accept-Encoding"] or '')..(ngx.req.get_headers()["Accept-Language"] or '')..ngx.var.msec)
                a_session_entry = "true"
            end

            ngx.header["Set-Cookie"] = {"a_vid="..a_visitor_id.."; Path=/; Expires=" .. ngx.cookie_time(ngx.time() + 63072000), "a_sid="..a_session_id.."; Path=/; Expires=" .. ngx.cookie_time(ngx.time() + 1800)}

            local args, err = ngx.req.get_uri_args()
            arr = {}
            a_query_args = "{"
            for key, val in pairs(args) do
                if type(val) == "table" then
                    table.insert(arr,'"'..key..'":"'.. table.concat(val, "|") ..'"')
                else
                    table.insert(arr,'"'..key..'":"'..val..'"')
                end
            end
            a_query_args = a_query_args .. table.concat(arr, ", ")
            a_query_args = a_query_args .. "}"


            ngx.var.a_visitor_id = a_visitor_id
            ngx.var.a_visitor_new = a_visitor_new
            ngx.var.a_session_id = a_session_id
            ngx.var.a_session_entry = a_session_entry
            ngx.var.a_query_args = a_query_args
        }
    }
}

This gives us log lines like this:

{"remote_addr":"127.0.0.1","remote_user":"","scheme":"http","uri":"/index.html","connection_request":"1","connection":"2","msec":"1578043737.950","pid":"3080","host":"127.0.0.1","server_name":"localhost","time":"2020-01-03T10:28:57+01:00","request":"GET / HTTP/1.1","status":"304","size":"0","query_string":"","referer":"","user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36","proxy_ip":"","visitor_id":"7c958ba7becea5f5bea54a80b899330e","new_visitor":false,"session_id":"c5d80b3d6f3f4706cac4eea0a8ad2f12","session_entry":false,"query_array":{}}

For a new user, it could look like this. Consider the session_entry and new_visitor keys:

{"remote_addr":"127.0.0.1","remote_user":"","scheme":"http","uri":"/index.html","connection_request":"1","connection":"1","msec":"1578141208.707","pid":"12660","host":"127.0.0.1","server_name":"localhost","time":"2020-01-04T13:33:28+01:00","request":"GET / HTTP/1.1","status":"200","size":"674","query_string":"","referer":"","user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36","proxy_ip":"","visitor_id":"b9323e1cd1379aeb504d40f0ce9c822c","new_visitor":true,"session_id":"607a9bce4d67e4d6c0e9b98848d7f9d1","session_entry":true,"query_array":{}}

Filebeat Configuration

To ship our logfiles to Elasticsearch we need to tell Filebeat where our logfile is, that it is in JSON format and where to ship it. For our project the config is as simple as, given that we want to organize our log data in daily indices:

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - PATH-TO-LOGS\access-tracking.log
  json.add_error_key: true

setup.ilm.enabled: false
setup.template.name: "custom-filebeat-tracking-logs-%{[agent.version]}"
setup.template.pattern: "custom-filebeat-tracking-logs-%{[agent.version]}-*"

output.elasticsearch:
  hosts: ["ES-HOST"]
  index: "custom-filebeat-tracking-logs-%{[agent.version]}-%{+yyyy.MM.dd}"

You need to modify the path to the log file, the Elasticsearch host and the index format. The format here gives us indices like “
custom-filebeat-tracking-logs-7.4.0-2020.01.03″.

Now we are able to track our users, generate some log files with user information and ship it to Elasticsearch.

In the next part we will start working with the data and build a nice dashboard out of it.

The post Building your own Web Analytics from Log Files – Part 4: Data Collection and Processing appeared first on https://www.frederikwerner.de/

]]>