Mailchimp Contact Segmentation is BROKEN

I have been testing Mailchimp as a potential email service provider for my newsletter lists. One goal was to test Mailchimp’s email deliverability against another provider. If results were good, I was planning on moving to Mailchimp for all list management. To start, I wanted to try to find a way to increase newsletter open rates by finding users who were not opening emails from one service provider and using another provider to see if that affected open rates. This could tell us which provider was more effective at delivering emails. I was indeed able to find a small group of users from another provider who never opened an email that are now consistently opening newsletters when sent via Mailchimp. As I begin to develop and optimize my workflow for all of this, I think I am running into a flaw with Mailchimp’s segmentation system. Either that or I am running into a highly confusing scenario and I may be missing something. You be the judge.

Mailchimp Audience Segmentation – Is It Flawed?

Mailchimp offers a very powerful tool that allow list owners to identify users who have either opened or not opened recent broadcasts. Let me share the numbers and you will see the issue in their segmentation calculation.

A current list I am managing has 1,256 contacts, all marked as active and subscribed. To keep the list fresh, I archive all unsubscribed users regularly.

First thing to note, the last 6-7 campaigns have resulted in an average open rate of 20%+. Here is the first segmentation using their drop down menus.

Campaign Activity = opened = Any of the Last 5 Campaigns
AND
Email Marketing Status = is = Subscribed

Here is a screen shot of the segmentation:

This segmentation returns 387 contacts. This makes sense given the average open rate is roughly 20% and the list size is 1,256 in size.

Now the confusion starts. I wanted to archive users who have not opened ANY of the last 5 campaigns. Why did I choose 5? I didn’t. It’s one of the solutions within the hard-coded Mailchimp drop down menu item.

Campaign Activity = did not open = Any of the Last 5 campaigns
AND
Email Marketing Status = is = Subscribed

Here is a screen shot of the segmentation:

This segmentation returned 1,221 contacts! This cannot be correct based on the chosen filters. How can the number of contacts who did not open ANY of the last 5 campaigns equal to 97.2% when the list is active and has an average open rate of 20+%?

One odd thing I did notice was that if I chose “did not open”, the segmentation for the last portion of the filter would default to “All of the Last 5 Campaigns” like so:

Campaign Activity = did not open = All of the Last 5 campaigns

Here is a screen shot of the segmentation:

Numbers for this segmentation makes a little more sense as it returned 869 contacts. Given open rates are at about 20%, 869 would be a reasonable number for users who did not open all 5 campaigns. It may be reasonable but I cannot confirm this is correct since it could also be including users who were added prior to the last 5 campaigns but for whatever reason, never received all 5 campaigns. To rule this out, I added another filter:

Campaign Activity = did not open = All of the Last 5 campaigns
AND
Email Marketing Status = is = Subscribed
And
Date Added = is after = a specific campaign was sent = (I selected the 5th oldest campaign)

This resulted in 581 contacts. ie. Mailchimp determined 581 contacts did not open all of the last 5 campaigns. Right away, I was wondering why adding the Date Added filter would results in such a difference number than without the Date Added filter. This number (581) may be precise mathematically, but as far being accurate at filtering inactive users, there is also no way to confirm this. It’s now saying 581 contacts (46%) of all active users did not open all 5 campaigns. Given an average open rate of 20+%, having 46% of all users who did not open all 5 campaigns does not really add up either. Again, there is no way for me to confirm this number.

So. Should I archive these 581 contacts? Absolutely not. Let’s say this number is indeed accurate in and 581 contacts did not open all 5 campaigns. What if a user opened 1 of the last 5 campaigns. I do not want to archive this type of contact. Remember, I am trying to find inactive users, not low activity users.

Mailchimp Support

Believe it or not, I have been in email contact with support. To be clear, I am testing their free tier (<2,000 contacts) and support is not available for free users. I did, however, find a sympathetic Facebook rep who was understanding enough to open up a ticket for me. Support reps were courteous, understanding but slow to respond, probably due to shortage of staff due to COVID-19. Maybe due to shift work, within the same ticket, I have gone through a few reps already and they have all confirmed the numbers that I am seeing but none were unable to clearly explain why the numbers do not make sense.

It has been suggested to me by one support staff to manually check each campaign and find users who have not opened emails, campaign by campaign. Doing this by hand and then having to cross reference email addresses after every campaign would require some serious stats analysis. Can I do this? Yes. Should Mailchimp users be expected to have the knowledge and capability to even know how to calculate and cross reference campaign results by hand? Definitely not. It does not make sense to have an email service provider NOT provide this feature since all of the data is already there. In this case, it is highly likely that the Mailchimp system does have a way to find inactive users (and the math is correct) but the calculation itself may not coincide properly with the actual words in the segmentation drop down filters (ie. any of the last 5 campaigns, all of the last 5 campaigns). Again, without knowing their segmentation algorithm, I cannot verify this.

Another suggestion was that I archive users who have a low reputation, say 2 reputation stars. This is quite ridiculous as I have no idea how Mailchimp determine user reputation.

One support staff who understood that I was trying to filter for “sent the last 5 campaigns and who did not open any of them” suggested I use this segmentation plan:

Campaign Activity = Did not Open = Any of the Last 5 Campaigns
AND
Campaign Activity = Was Sent = All of the Last 5 Campaigns

Here is a screen shot of the segmentation:

The result of this brought back a little over 500 contacts. This number seems about right BUT… when I check the details of the first contact from this segmentation, it clearly shows that this contact has plenty of OPEN and CLICK activity! There are actually more contacts listed who also have open and click activity. How can Mailchimp’s segmentation be this inaccurate? The above segmentation should have identified users who have NOT opened any of the last 5 campaigns but instead returned contacts who were active and engaging to our newsletters. I am shocked to see this type of error in the Mailchimp segmentation system. Based on this alone, I can confidently say that Mailchimp’s segmentation system is clearly broken, at least for my account.

It seems tech support does indeed understand the issue at hand and have knowledge on how to accomplish certain tasks inside Mailchimp. But they seem to be lacking the knowledge on WHY something should be done and why the system is inaccurately reporting for INACTIVE vs ACTIVE users. That’s my take on all of this.

I have yet to to find a solution with support that can allow me an easy way to identify users who have not opened any of the last 5 campaigns (who I assumed, has also received all 5 campaigns) so I can archive them.

Mailchimp Audience Segmentation – Conclusion

Given Mailchimp’s segmentation system seems a little confusing and not clear on how segments are actually filtered, I do not have confidence I am archiving the appropriate contacts. It may be that wording from the drop down menu items do not match the math behind the calculations. I am just not sure at this point.

Maybe it’s my methodology or my lack of understanding Mailchimp’s criteria for segmentation. But it is clear that their support staff up until the time of publishing this post, were not able to find a solution either. If anyone can figure out how I can accurately archive inactive users (ie. users who are not opening our newsletters) but who have been receiving them, please let me know. I was hoping I could keep using Mailchimp to manage and grow our lists. But at this point, I just don’t have the confidence to do so.

If you use Mailchimp segmentation, I would suggest you run the above filters and check your own results. I still cannot believe my Mailchimp account erroneously identified ACTIVE users as being INACTIVE!

What am I missing? If you have any suggestions, please comment below.