098: Is Wix the Best CMS for Building Accessible Websites? — BomBombs Hot Chocolate

By /

In this episode, we dive into the recent study released by the HTTP Archive claiming that Wix is the most accessible CMS on the market in 2024, based on aggregate testing data from Google Lighthouse.

Listen

Watch

YouTube video

Mentioned in this Episode

Transcript

Chris: Welcome to the Accessibility Craft Podcast, where we explore the art of creating accessible websites while trying out interesting craft beverages. This podcast is brought to you by the team at Equalize Digital, a WordPress accessibility company, and the proud creators of the Accessibility Checker plugin. And now, on to the show.

Amber: Hey everybody, it’s Amber, and I’m here today with Chris.

Chris: Hey everyone.

Amber: And Steve.

Steve: Hey, everybody.

Amber: And we are going to be having a beverage and talking about a new report that came out. If you want to get full notes or a transcript for this episode, you can find them if you go to AccessibilityCraft.com/098. I am very excited this week because I think we’re making up for our horrible beer that we had to drink last week with something sweet and wonderful. What are we drinking today, Chris?

Today’s Beverage

Chris: We are trying, across the three of us, some assorted flavors of BombBomb’s Hot Chocolate. So, I think these originally like went viral online because they made these hot chocolate bombs where it was like, there was a middle and then it was encased in chocolate.

But in the middle were like marshmallows and other things like candy cane bits and whatever else. And you pour the hot drink over it, chocolate melts, et cetera. I could not get those shipped in time because of the proximity to the Holidays, but they did have an alternative like premium powdered hot chocolate that came, I think in like 12 different flavors.

So I got us those and I’m trying and they kind of, all the little flavor powders came in these cute little cups. So I’m trying pistachio macaron today.

Amber: Steve’s got the whole box.

Yeah. He’s going to be Vanna White. They, so the cups look like mini coffee cups, but they’re like shot size, but they’re not actually. They’re smaller than the quantity you’re supposed to mix. So you wouldn’t actually mix in it. It’s just decorative containers. But I I’m picking coconut cream. What are you going to drink, Steve?

Steve: Well, what did Chris say? What was his?

Chris: Pistachio Macaron.

Steve: Okay. Okay. So I’ve got all of them. So I get to pick here live on the podcast.

Amber: So what are some of the options that are jumping out at you?

Steve: The Peppermint Swirl is standing out. The Rocky Road. The there was another, what was it? Salted Caramel Donut.

Chris: Oh, yeah.

Amber: Now, wait just a minute. Can it really be Rocky Road if there’s no chunks in it? Like, Rocky Road has marshmallows and walnuts and you have to chew it.

Steve: Maybe it has, I don’t know.

Chris: Maybe it’s chunky.

Steve: It’s like Rocky Road flavored probably.

Amber: Yeah, it’s got like a walnut marshmallow flavor in there with the chocolate.

Steve: Yeah, so my gut’s telling me to go with the Salted Caramel Donut, but since we’re coming up on the holidays, I think I’m going to try the Peppermint Swirl.

Chris: There you go.

Amber: Okay. All right. So I had to be very exact ’cause I’m a particular person. I got out a liquid measuring cup and I measured exactly six ounces of water .

Steve: Oh, is it?

Amber: Because I’m like, I wanna get it the way they say to do it. So I’m gonna pour this in my mug here.

Chris: And is your water still pretty warm?

Amber: It seems pretty warm still. My, so my initial reaction on this is, if you’re used to a hot cocoa mix, I can actually see more granulations of sugar in this. Like, if you get, I don’t know, like Swiss Miss or something off the store shelf, and I’ll be honest, it’s been a long time because we don’t normally buy that, we make our own.

But like, I think it’s more uniform in consistency, whereas this looks more like chocolate and sugar mixed together, which is more of what I would see when I make it at home. So that makes me think it might be good. I don’t know.

Chris: I will say my pistachio one smells pistachio-y, so that’s maybe a good sign.

Amber: Yeah. Oh yeah. This has a good coconut smell on it. I love coconut.

Chris: I was a little, I was hoping that there would be like little, when I saw the packaging, because I didn’t read very carefully, I was hoping there would be little baby hot cocoa bombs inside of these, like little small spheres. But it was the powder, so I was a little bummed about that, but I am excited by the smell.

So while y’all mix, I might give mine a taste because it’s, torturing me with the aromas.

Amber: I really want to know if yours is like as good as the chocolate pistachio things you had in Italy.

Chris: Oh no, absolutely not. It’s not. Absolutely not.

Amber: Italy wins.

Chris: But I will say as far as hot cocos go this is probably up there for me on like a powdered mix scale.

Like, because I just mix pure water in, so it actually has a little body to it. It’s not overly sweet. I get the pistachio. I get chocolate. And the chocolate’s even a little bitter. What do you think?

Amber: Yeah, I don’t love it, but I think it’s because when we make hot cocoa at home, we use milk. Well, I mean, we use oat milk.

We don’t actually use cow milk, but we use like something that’s not water. But it makes me think maybe if I had not followed their instructions and I had heated up milk and mixed it in, like, maybe I’d like a little more. It’s just like, it’s a little too thin for me. I don’t know.

Chris: I feel like Swiss Miss is definitely waterier when I have that.

Like I, I tasting this, I do feel like it’s a step up from a standard hot cocoa mix. But yeah, it’s not the same as like melting the chocolate yourself. What do you think, Steve?

Steve: It’s very pepperminty. Very festive.

Amber: Too much or it’s a good amount?

Steve: It’s just right. Tastes great. I like it. It’s a lot better than pickle flavored beer. That’s all.

Chris: Lowest bar ever.

Amber: You will choose this any day. I know. It’s like, if we get a drink worse than that ever, we will just have to stop and it will be the Accessibility Podcast without the Craft.

Steve: Yeah. No, I like this. This is this is great. Makes me feel very Christmassy.

Amber: Does it, so, you know, there’s like all these different kind of flavors of mint.

Is it like a candy cane mint? Like if you had stirred up a hot chocolate with candy cane, or is it a different kind of mint flavor?

Steve: Yeah. It’s like a candy cane. Like you pulled it right off a tree.

Amber: They grow on trees, right? They grow on Christmas trees?

Steve: Christmas trees. Yeah.

Chris Tells a Dad Joke

Amber: I almost want to put Chris on the spot because his mug says that we’re supposed to ask him about his dad jokes.

Chris: Do you want to, do you want a dad joke?

Amber: I want a dad joke before we start talking.

Chris: So a shout out to our developer, William, who owns DadJokez.com, with a Z, which I opened in a tab on my computer because I bookmarked that.

So, coffee has a tough time at my house. Every morning it gets mugged. That’s my dad joke for today.

Steve: Speaking of mugs, right? Yeah, Amber, you’ve got a special mug, right?

Amber: Yeah, well, this is my, I bought this during COVID. It says, you’re on mute in a speech bubble. It’s perfect for Zoom meetings, especially in the beginning of COVID when we were Zooming with a bunch of people who had never Zoomed before and weren’t familiar.

Steve: Yeah, and it’s ironic when Amber’s the one on mute.

Amber: Yes, that has happened. People would be like, Amber, where’s your mug? Read it.

Steve: Yeah. I don’t have a special mug. Oh,

Amber: I forgot to put the name up for anyone who’s looking. It’s BOM, spelled B O M B S, all one word, hot chocolate. And I think we would say thumbs up if you’re, yeah, thumbs up if you’re looking for a different kind of hot chocolate that’s not just a store on, on the shelves, regulars.

This might be worth checking out and get it on Amazon, Chris?

Chris: Yeah, that’s where I got it. I think you can get it off their main website too. And what this really does is it makes me curious to try their, the thing that really, they really like put them on the map, virally, the actual chocolate spheres.

Maybe we’ll have to try that in a future holiday episode. Maybe BomBombs will make a comeback.

Steve: Yeah.

The HTTP Archive Says Wix is the Best CMS for Accessibility

Amber: We can have hot chocolate around Valentine’s Day. So I think you can put that on your list for February. So we are drinking a sweet beverage. We’re going to see how sweet we’re going to be on this episode. Because what we are going to be talking about today is Wix, the best CMS for building accessible websites. Is it? Should we all abandon WordPress and switch over to Wix because we want things to be accessible?

This I was inspired by this after seeing an article that was published on December 4th in Search Engine Journal covering a recent report from the HTTP Archive that said that Wix was the best platform for building accessible websites.

The HTTP Archive has been doing this report for a number of years. They scan over 16 million websites using Google Lighthouse and that of course is based on Deque’s open source axe core library, which we’ve talked about in other podcast episodes. It’s the same one that Accessibility Checker uses, but we’ve modified to improve and make better.

And they analyzed over 16 million websites. They included, by scanning the front and trying to figure out what CMS platform the websites were built in, a ranking by CMS. And they said that Wix was the best with an average score of over 94 percent accessibility in Google Lighthouse. So we’re going to talk about that because of course, for all us WordPress folks, that’s very triggering.

So I want to know, Chris and Steve, when you first saw this study, when I shared it with you, what was your immediate gut reaction without reading it, and then we can talk more about diving into it.

Chris: Skepticism in a word and wanting to understand methodologies and how they actually approached it, because you never know. It could be true. Yeah. At least reading the headline, right?

Steve: Yeah. My initial response was, well, it, I will say my initial response may be a little different than my final conclusion. But my initial response was definitely like, yeah, right. But now that I’ve read through some things, I might have a little bit of a different take on that.

Amber: Yeah, I mean, I will totally admit I saw this. Well, so I saw it on Twitter, but I saw it on Twitter because somebody in the WP Minute Slack tagged me and they shared the thing from Twitter and they’re like, Amber, what do you think about this? And my initial was, what? No way. It’s a lie. I could not believe it.

Having recently spent some time looking at Wix and some other CMSs, but yeah, it is interesting looking at the data.

The Pitfalls of Relying Solely on Google Lighthouse Data

Chris: So one thing that immediately jumped out as I looked into this and I’m sure as y’all looked into this was the, what they’re judging the accessibility off of is Google Lighthouse and my, there’s been internal discussions around the reliability of Google Lighthouse and if that is actually a good measure of accessibility, but I don’t think we’ve ever talked about that publicly in a meaningful way. So do y’all think that Google Lighthouse is a good measure of accessibility?

Amber: Steve, I’ll let you go first as the tech, our tech person.

Steve: I mean, I think Lighthouse, like any other tool, right? You have to almost evaluate the tool itself. I know internally, we don’t rely on Lighthouse to do pretty much, I mean, if I’m in there doing a performance page speed scan, I might check the box for accessibility just for my own interests. But and this is not scientific in any way, but what I have noticed is it seems to be very lenient and those scores always seem to be pretty high. It seems to be pretty passing.

And now, I mean, Lighthouse does provide you with a list of things to manually audit and stuff. So, but, I don’t have a lot of scientific studies into why I don’t use it or why I don’t particularly think that it returns back a lot of valuable information for me. But that’s just kind of my feeling on it.

And I just, I, the word’s probably dismissive. I’m a little dismissive of Lighthouse and it may not be for the fact that Lighthouse is wrong, but it may be that I just have better tools at my disposal.

Amber: Yeah I would choose our Accessibility Checker or WAVE or maybe even the Axe browser extension itself.

Steve: Yeah.

Amber: Over Lighthouse any day. I also feel like I have never seen a website score below maybe a 60 on accessibility in Lighthouse, including ones that are just horrendous.

Steve: Yeah. Yeah. I mean, I think that’s what I’ve, what I was getting at. Like I’ll, we’ll be doing a on it remediation on a client’s website and I’ll run it and I’m like, that’s awful high, but we’ve got all these issues we’ve identified and or that other tools have identified, but yeah it’s definitely on the high end. I don’t know why.

Amber: Yeah I will say I very recently we put out our call for speakers for WordPress Accessibility Meetup and somebody reached out and said, hey I’d be interested in speaking about how we got a website to have a 100 percent accessibility score in Lighthouse.

And I went and looked at the website and it was really bad. Like it had failed color contrast. It was missing focus outlines. It was just, there was a lot about it that wasn’t accessible. And I messaged him back and I was like, you know, I took that as an opportunity to say, hey, here’s some things that I noticed.

And what I suggested to them was, I would love to have you talk, but what I would love to have you talk about is I’d love to see you go back and continue working on this website and then give a talk about what Lighthouse missed.

And how did you go beyond Lighthouse, right? Once you are able to get the website remediated.

I just, and I feel like I’ve seen a lot of those. Where it has a good score, but it’s not accessible.

Steve: Yeah. And I think in the prep of this episode, some of the stuff that I was reading through is what kind of changed my, kind of my knee jerk reaction to Wix being the most accessible was this one article that Amber put in our notes to kind of go over was building the most inaccessible site possible with a perfect Lighthouse score.

So this is a, we’re linked this article in the show notes, but basically the author goes through and it creates basically a website that’s pretty much unusable to, to any screen readers and, but it still maintains the Lighthouse score. And I think he does this in an effort to to underscore why manual auditing, not just of the site, but of the tools that you’re using is important.

But also how if there’s a platform that’s putting out theme templates that you can utilize to build a website, right? There’s ways that they could game the score and make it higher. So, I mean, just to kind of briefly go through what they do is it goes through and they try to, he calls it an exclusion first website, so it’s completely inaccessible.

So, so he basically just rolls up a basic website with some content. And then he puts a hidden attribute on the body tag, right? Which basically hides it to the whole website, to a screen reader. But not only does he do that, he also binds it to, to being initiated by CSS and JavaScript, the display. So for sighted users that aren’t using a screen reader, right?

He turns the site back on to being visible to us, but he does it only with JavaScript and CSS. So we can see it. It looks totally normal, but screen reader sees nothing. So, and then at every step it goes through and he runs a Lighthouse test and it shows that it passes, right? Screen reader users, he actually will do an ARIA hidden on the body tag to hide the whole thing.

And he acknowledges that Lighthouse actually updated this. So, but he found that I’m assuming the author is a he, but the author goes through and adds a tab index of negative one, which then breaks his Lighthouse scores back to a hundred percent. So it goes through and removes focus states, like no focus states at all.

And…

Amber: Which is so common. That is one of the biggest issues.

Steve: Of all websites. Yeah.

Amber: Yeah.

Steve: When really, if you just leave it alone, the browser will natively handle it, right? And and that’s what we do in the Accessibility Checker plugin. We have a fix in the settings that where you can actually restore the browser defaults for your focus state.

Amber: It, this reminds me a little bit, like what you were saying that it’s easy to game those scores.

Steve: Yeah

Amber: Because we were talking about this at team meeting because we audited a website recently where all the forms were built with divs and they hadn’t been remediated with proper ARIA.

Steve: Yeah.

Amber: And so the fields, if you were, which aren’t really, but they were, even if they were like focusable, there were unlabeled. Because the, you know, the labels were just text, not connected to it in any way. But we were talking about, you know, this probably wouldn’t flag on any Accessibility Checker as unlabeled fields, because Accessibility Checkers don’t look, ours or WAVE or AXE or whatever, Lighthouse, don’t look at every div and say, oh, this is probably, right?

So, so it can look like, it’s good, even though the underlying HTML is just very poor.

Steve: Yeah, the automated tool doesn’t know whether or not to evaluate it because it looks like just a div. And then, and like you said, the automated tool is not going through and evaluating every div for the potentiality that it is a form. Right?

So, so I think through this through this article, that’s what he’s doing. He’s going through and he’s over and over, like he’s breaking it, the functionality. And making sure that like a sighted user can actually still see the site, but screen user can not exit. And he goes as far as to like break it for like high contrast mode, basically takes a high contrast mode, ’cause it’s just a media query, right? And basically just sets it to, to the background. So like it’s black text on a black background. Lighthouse still says 100%, right? Like, and so he does this over and and just to, you know, prove a point that this could be gamed. So my reading of this started to kind of attach it to the Wix.

Well, okay, well, Wix is not, you know, we’re going to get into like the differences between Wix and WordPress and all that, but like, if somebody wanted to like tout our themes or all of our themes have the highest Lighthouse score possible, there’s always ways to game it. And I don’t want to just call out Lighthouse. Like I’m sure there’s ways to game WAVE or there’s ways to game the Accessibility Checker, right? .

Amber: Yeah, I mean we’ve seen like with WAVE, that’s something that some of the overlays have gotten in trouble with is they actually block WAVE.

Steve: Yeah.

Amber: From being able to scan because they don’t want people to be aware of the problems.

So there’s definitely ways to game it. I think we should take a quick commercial. And we’ll be right back.

Brought to you by Accessibility Checker

Steve: This episode of Accessibility Craft is sponsored by Equalized Digital Accessibility Checker, the WordPress plugin that helps you find accessibility problems before you hit publish. Thousands of businesses, nonprofits, universities, and government agencies around the world trust Accessibility Checker to help their teams find, fix, and prevent accessibility problems on an ongoing basis.

New to accessibility? Equalize Digital Accessibility Checker is here to teach you every step of the way. Whether you’re a content creator or a developer, our detailed documentation guides you through fixing accessibility issues. Never lose track of accessibility again with real time scans each time you save.

Powerful reports inside the WordPress dashboard, and a front end view to help you track down hard to find issues. Scan unlimited posts and pages with Accessibility Checker Free. Upgrade to Accessibility Checker Pro to scan your website in bulk, whether it has 10 pages or 10,000. Download Accessibility Checker today at EqualizeDigital.com/Accessibility-Checker. Use coupon code AccessibilityCraft to save 10% on any plan.

Platform Rankings and Percentage Scores from the Study

Chris: So while the commercial was running, I was thinking about this study and I think it would be helpful to just kind of lay out how the HTT P Archive ranked builders, even though there may be some questions around Lighthouse or any checker, you know, being the perfect measurement for judging the overall accessibility of a platform, right?

There’s definite questions there, but here’s how they ranked and what the percentages were for the quote, most accessible CMS in 2024. Wix, and there’s quite a few of these Wix got 94. Squarespace, 92. Google Sites, 90%. Duda, 87%. HubSpot CMS, 87%, Adobe Experience Manager, 87%. And Contentful, 87%. PixNet, 86%. Weebly, 86%. Sitecore, WordPress, and GoDaddy Website Builder all got 85%. And then Webnode, CraftCMS, And Contao and Drupal all got 84%. Liferay 83%, Tilda 83%, Typo3, CMS 83%, and Gemdo 83%.

Amber: Yeah. You know what just jumped out at me just now as you’re reading that list? Shopify isn’t on here. Doesn’t that seem weird?

Steve: Well, is that because it’s an eCommerce? Like… I mean, they could be below that 83%, right?

Amber: Oh, maybe. They’re worse.

Steve: Who knows?

Amber: It just seems a little weird to me. I think that Shopify is not on here because I would think that if you’re scanning 16 million websites. So I did know when I read this, that they said that they only included in the averages, what power, what they could tell powered more than 10, 000 websites, but I’m certain Shopify is in that category.

Chris: It absolutely must power more than 10, 000 sites. So maybe it’s a score issue. I don’t know. You know, the other thing that I was reading about as I was preparing for this about the Lighthouse versus Axe Core where it pulls its rules is I learned that Axe Core has around 80 rules, I think, and Lighthouse is only using 50 of them.

So that may also introduce some limitations into this scoring, depending on how they chose the rules and why they omitted I mean, I don’t know what that is, like 30 percent of their rules or 40 percent of the rules.

Steve: I mean, they also have some of their own as well. They’ve got a handful of their own evaluations as well. We’re similar. I mean, the accessibility checker doesn’t pull the Axe Core wholesale, right? We kind of pick and choose, from it, what we want, and we go a step further, right? We curated it a little bit. Sometimes we collect them into one rule and sometimes we extend them. So we’re modifying them as well, but yeah.

The Study Results: Do We Agree?

Chris: Most aren’t taking it at face value. That’s definitely true. Amber, you’ve been doing manual auditing for native components of several CMSs, including some of the ones on this list. How would you say your findings are aligning with this in general, with the work you’ve been doing?

Amber: Yeah, I, I actually want to I’ll give a little sneak peek of this.

I have a spreadsheet. It’s similar to the one I did for my page builder and WordPress comparisons. There’s going to be more of this coming out because there’s a YouTuber who asked me to create this. And she built out sites in Webflow, ShowIt, Wix. Squarespace and Shopify and just put native components on there.

And I went through and tested a lot of the same kind of things that I tested for that page builder. So navigation, she wanted to do like an, like a hero where it’s an image with text kind of banner embedded videos. She wanted to do accordions, carousel, contact forms. So that’s a little different. I didn’t do them in page builders because I think you should use a form plugin, but of course if you’re building this other way, you’re pretty much using the CMS’s forms. And then I looked at it overall, and I’ll just , say it right now. I did not find Wix to be the best when I looked at these components. It was, out of the five that I tested, it was number four. So, it was not the top.

Shopify was number one. So, it’s a, I really find it odd that Shopify is not on this, and I almost want to, like, dig into that data. Like, why is Shopify not on the HTTP archives data. Squarespace was number two. So that aligns with what you just read out, Chris. And then Webflow was number three and ShowIt was number five after Wix.

So of course, you know, not total overlap with this, but it seemed kind of, I saw the same thing. Oh, look, I don’t even know what, View, we’re supposed to be on. Nope.

Chris: We’ll get one eventually. That’s right.

Steve: It’s our fourth guest. The spreadsheet.

Amber: There we go. That spreadsheet is just going to sit there.

But no, so, so I’d been doing it and I had been spending a lot of time doing manual testing of components. And, and this is a thing we talk about all the time. Automated tools are fabulous. They can really help you speed things up, particularly when you use a tool like Accessibility Checker that can crawl your whole website.

But using an automated tool does not mean that you don’t have to do manual testing. Because there’s a lot, like for example, accordions is a great example. I found this both on the WordPress page builders and I found it on Wix. If I go back and I look at this, they’re not using headings inside button tags.

They don’t have ARIA controls. You can style something to look like an H3, but it doesn’t actually apply an H3 tag to it. In there when you’re, it has keyboard traps, like it overtakes the arrow keys because it just wants you to use the arrow keys to move between the accordions. So you can’t actually scroll the page if you’re a keyboard only user if the accordions have received focus.

There’s just a lot of weirdness around that, but it can only be found with a manual test. So, you know, I still kind of, it’s hard, like when I see this report about Wix. I’m like, I don’t know if I believe it having tested the manual thing. Of course, there’s a lot of it, right? If you didn’t put that accordion on your website, then maybe you’re fine.

Reviewing a Wix Showcase Site with a 100% Lighthouse Score

Steve: So do we want to tear apart the example site from Wix here?

Amber: Yeah let’s do it. Do you want to share your screen for a minute, Steve? I know we don’t want to make this too long of an episode, but I thought it would be interesting. I went through Wix’s. Showcase. I’ll explain this while you pull it up.

And I found some, a site that was in their showcase, that has a 100 percent accessibility score in Lighthouse. And I feel like in maybe three minutes, we could real quick just flag some problems with this site. I mean, when you look at it, what do you notice, Steve?

Steve: Well, one, let me put the browser in the, put the shields down.

Amber: It has some really janky animations that I’ll tell you don’t respond if you have your computer set to preferred reduced motion.

Steve: Yeah. So, I mean, I am noticing a few, InstaFeed console errors off the bat, but that’s not necessarily accessibility problems, but the, do we want to run Lighthouse? Is this how we should start?

Amber: Sure. So how would someone run Lighthouse if they’ve never done that before?

Steve: So if you’re using a Chromium browser, you should be able to open the inspector and then go to the Lighthouse tab, and then you’re presented with you’re presented with some options here. So I’m going to go to categories and only select accessibility because that’s all we care about right now. And then we could do desktop. That may be more favorable. Does, did the study say that it ran desktop and mobile or desktop? Like, does it specify?

Amber: So I think they I think on the performance category, they looked at mobile, but I did not see it specified for, did you, Chris, see if it said?

Chris: I see that it has It has accessibility scores output for both desktop and mobile separately based on the tabular data I’m looking at right now.

It’s possible I’m looking at the wrong tab.

Steve: So we’ve run a, I’ve run a Lighthouse test for accessibility and it does come back with a hundred percent on desktop which is what we expected. Now…

Amber: Can I point something out that I can see right now? Maybe it’s what you were going to say. Go for it.

Steve: So, so for people listening, there’s like a, I don’t know what this is, but the very first section below the header It’s like it’s it’s , two columns left is yellow, black background with blue text. And the right is a blue background with white text. And to run the Lighthouse, you know, I opened the inspector, which kind of pushes the viewport in and makes the viewport smaller. So when you, your viewport gets to about, a little under 1200 pixels wide. The white text on the blue background starts to overflow onto the yellow background on the left side. So like the, and if I keep going as you’re noticing.

Chris: Yeah. So this is what I was gonna say. Zoom into about 300% without your inspector open. And that’s another fun one.

Steve: Is this framed in? Is this website framed in?

Amber: Yeah, I don’t know. I noticed that it doesn’t look mobile responsive at all.

Chris: Yeah. So if you zoom in to like 200, 300%, it’s just, it’s totally…

Amber: Well, it’s got a sideways scroll and it’s and text overlaps is completely unreadable. And so this right here, for someone who has low vision, many people might be looking at your website zoomed to 200 percent or more. So clearly this is not, no thought has been put into low vision users in that section.

Chris: I just want to say too, that I did did run a check on it. And the other thing I noticed was I think there’s not even an H1, like there’s no H1 anywhere.

Steve: Good catch. Let’s run. So we use an extension called Headings Map and Chris is 100 percent correct. You, you earned yourself another cup of cocoa. There is no H1.

Amber: Talking about gaming the system I haven’t figured out yet why it is this way, but if you even run WAVE on this website, it’s the same thing. Neither WAVE nor Lighthouse find any color contrast issues. On this website. But, when I look at, there’s a section where there’s a heading. Oh, this is hilarious.

In your, you can see this in the headings map. Community Collection is supposed to be one heading. But it’s actually two separate H2 tags. Cause I guess they didn’t know how to like, put it in the same H2 tag and have a line break. But the text there is white on a very light blue background. I use TPGI has this really handy desktop application for you can sample anything.

And I use it a lot of times in Figma files and stuff like that. But I was like, okay, WAVE can’t see it. Lighthouse can’t see it. I know this fails. And I checked. Yes, indeed. That’s a 2. 6 to one color contrast ratio.

Steve: Yeah. Which was it?

Amber: The, well, the white text on the light blue background, but probably all the text in that whole section.

Yeah. But neither, none of those automated tools can find it. And so it says 100 percent in WAVE for color contrast, it says 100 percent in Lighthouse, but it’s not, it fails.

And I don’t know why. And again, this like really highlights why you gotta test stuff, but also to be totally honest, like sometimes you just have to know, like I’m looking at that and I was like, there’s no way that passes.

Steve: Yeah, totally. And in the headings map, there’s a star rating. There’s a five star rating. So no context.

Chris: What would that sound like on a screen reader?

Steve: Oh, it would probably leave

Chris: Star?

Steve: No, I don’t think it would.

Amber: Yeah, it might, depending on what it is.

Steve: Yeah, they are.

Amber: They’re emojis.

Steve: Yeah.

Amber: It’ll say like, heading level two, image, star, image, star, image, star, image, star.

Chris: That’s some solid content right there.

Steve: So, so I will say in, there may be something up with the color contrast. I don’t know what it is. I did see that the, that white text you were talking about, Amber is using RGB, is using RGBA or RGBA to define it instead of just a hex.

I don’t know if that’s throwing it off, but like, so. Most people don’t know this, but the Accessibility Checker has a weird way of scanning websites, it’s not even running on, but I wanted to see if we actually pick up that color contrast in the accessibility checker and we don’t, that’s it.

Amber: So this is probably an Axe limitation.

Steve: So, so that’s an interesting one. Now, the, we do flag an empty button that the other two that Lighthouse and WAVE were not flagging. Well, WAVE may have…

Amber: No WAVE said zero errors. Okay. WAVE also thinks that this is a pretty handy, nice little website.

Steve: Yeah. Yeah. So. It’s returning this as an empty button.

Amber: I don’t know. There is one depending on where it is. I see one, but does it have an ARIA label on it?

Steve: It does.

Amber: Oh.

Steve: Oh.

Amber: We need to go look at our rule.

Sometimes we find things and then we enhance them.

Steve: Although, okay, but this is very interesting and I think this is good. It has an ARIA label. But when we do, in our evaluation, if you look at the code snippet, does it have the ARIA label? Hold on. No, it does not.

Amber: So they add the ARIA label late.

Steve: Yeah.

Chris: So the ARIA label was in the wrong spot.

Steve: Yeah. So the Accessibility Checker does, it does a mix of what we call PHP and JavaScript scans, right? So some things are evaluated with a fully rated page and some things are evaluated with parsed HTML off the page, and this is for performance and for some legacy reasons, but that ARIA label is actually added late in the process from the front end via JavaScript.

Amber: So there’s some possibility that for some users it may not load.

Steve: Yeah, totally. So, if JavaScript is disabled, then this ARIA label would not show. If JavaScript was disabled, a lot of stuff probably would not show on this website and most websites, to be fair.

Amber: I mean, I think honestly, if I were looking at this website, like the worst things to me are the lack of mobile responsiveness, the overlap, the color contrast.

Steve: Yeah, we can look at that real quick. I mean, real quick, let’s see. Do you want to do a phone or you just want to look at it? Like it definitely, there’s something really odd.

Chris: Yeah.

Steve: I wonder if I refresh. Yeah, there you go. So what’s going on is they’re not re initiating. They’re not pre instantiating all of their JavaScript on viewport resize. So there’s a lot, I don’t know, I don’t know how this is made, but there’s a lot inside this theme that must run on JavaScript. So when you size down it just, you know…

But if I do a refresh, this is going to make all the JavaScript reinitialize and calculate based on the current viewport width.

Chris: Would they do that for performance reasons?

Steve: Although it’s not working. Why is it not working? Oh, I know why. It’s because it’s specifically targeting a mobile device.

Everyone: Oh…

Amber: Which is silly because how many of you, raise your hand on this podcast, like to look at Windows side by side sometimes on your desktop computer? Right? I think we all do that.

Chris: And not everybody has like a 4k screen where it’s, you know, the equivalent of two 1080p monitors or four squished together.

Amber: Now you’re just bragging, Chris.

Steve: So, so if I emulate like an iPhone SE, right, which is one of the smaller iPhones left these days you can see that it’s fine if I load it and all the JavaScript initializes and resizes based on that viewport.

But like, if I’m not emulating a device, right. Like, it’s kind of funny the way it looks when I turn it off, but because it’s still mobile and it shouldn’t be until I refresh.

Amber: So I think we have sufficiently shown that a website that gets a 100 percent on Lighthouse is not necessarily an accessible website.

Let’s go back to that study. What what are you thinking, Chris? I think you had a question queued up here.

Does WordPress’s Rank in the Study Make Sense?

Chris: Yeah. Well, I think the next question was more around just WordPress and its overall position in this and in this list, right? So it’s middle of the pack, right? If you’re judging solely based on these lighthouse scores from this aggregate data. Do we feel like that’s an accurate or fair representation based on what we’ve seen or not?

Amber: So here’s one thing that I would say that I was thinking about this is, so first of all, there are a lot of builders here that I’m not familiar with. So it’s really hard for me to judge how WordPress compares to Contentful when I’ve literally never looked at Contentful or even heard of it until looking at this study, right?

But one thing that did occur to me is that there are some CMSs that are very limited in what you can do with them. And so you can insert their accordion component or their carousel component or their tab component And that is the one you get. And so it is possible that if they have put a lot of effort into accessibility, and those are the only components you use, then they might very well be much more accessible than WordPress websites, where you can choose from one of 500, I don’t even know.

There’s probably more than that. Accordion plugins. So I don’t know. Steve, as a developer, what do you think?

Steve: I mean, on my takeaway of this study, like, do I think that the description…

Amber: Is it right that WordPress is in the middle?

Steve: Well, I mean, the spread’s not super, it’s not, like, what are the percent, what’s the spread between Wix and WordPress, Chris?

Chris: It’s an 11 percent spread across all of these, between 83 and 94%.

Amber: And it’s 9 percent difference in Wix and WordPress.

Steve: Yeah. So, on one, I think you’re right, Amber. I think what you’re dealing with is you’re dealing with an open platform and you’re dealing with a closed platform, right? Like Wix, now forgive me for not being very versed on Wix because I don’t typically use it because I’m a WordPress guy, but I suspect that all the themes available or components available to Wix people are created by Wix the organization, correct?

Is there a theme library?

Amber: I don’t think so. I know like Shopify and Squarespace has a really big like third party developer marketplace, but I don’t believe, and listeners please comment on this YouTube video, and tell us if we’re wrong. I don’t think that’s the case for Wix.

Steve: Yeah, I don’t think so either. I think it’s closed and I think it’s handled by a, you know, a team and they probably have development processes in place and they probably have an initiative in place to ensure that the Lighthouse results are high, right? And they can control that. In WordPress, WordPress is open. Third party, like it could be third party. It doesn’t even have to be approved by WordPress. org. Like I can make a theme and sell it, Theme Forest or, you know, like back in the day or on my own website, right? Like I can sell it independently.

So I think WordPress is a little more fragmented in that sense. And that probably does lead, and in all fairness WordPress has a low barrier of entry, right? You don’t have to have a ton of web skills to, to be able to make a WordPress theme, at least traditionally you didn’t. The block, full site themes are a little more complicated, but so you may be dealing with developers with, you know, a much more junior skillset or even an entry level skillset.

So, those numbers probably, they don’t surprise me a ton in the fact that WordPress is down a little bit from Wix. Now, I will say that I don’t agree, I don’t agree that any of them are that high.

Amber: Do you think they’re all too high?

Steve: Yeah, I think so. I mean, I think, and relying on one tool maybe may not be the best way to go and not having any evaluation on if anything is being done to circumvent or to game those outcomes, right? I think kind of might muddy those numbers. I’m not saying that anybody’s acting nefariously, but so I don’t know. I’ve seen a lot of WordPress websites and I’ve seen some really unaccessible WordPress websites.

So it’s not beyond reason that those, not saying those percentages are right, but maybe that order might be somewhat right based on what WordPress is and what Wix is.

Chris: And just to echo what you’re saying a little bit, I think there are probably are some advantages built into having a smaller share of the web market in general, because with that smaller share and also maybe, if you’re a closed system, that’s like a double whammy of having increased control over the outcomes. The larger, the piece of the pie you’re holding and that you’re trying to, you know, control in terms of code or accessibility or whatever the outcome is, the harder of a time you’re going to have to hold down to that. And then if you add open source on top of that, I wouldn’t say I’m particularly surprised to see WordPress in the middle, personally.

Amber: Yeah. You know, what I told someone the other day was with great power comes great responsibility.

So if you have the ability to edit all the things, you need to make sure they are accessible. So, but as we’ve seen, supposedly here’s a really great example of a Wix website. Because it’s from their showcase. I don’t think they put it in their showcase if they didn’t think it was awesome and a great example of what you can build with their platform.

And so clearly some attention needs to be paid on that. And I mean, we probably all do, right? No matter what CMS you’re building, you need to really test yourself and don’t just rely on Lighthouse or other automated tools.

Steve: And don’t rely on this report to give you comfort that your website….

Amber: Yeah. So, so you’re not going to be switching to Wix tomorrow, Steve?

Steve: No. And for many reasons and in all fairness, I haven’t used Wix. I’ve been in there. Years ago with clients, I think there’s been some migrations from Wix to WordPress that I’ve dealt with, but I’m not a Wix power user, but I think there’s reasons not to use it beyond accessibility, but also don’t… I wouldn’t find comfort in a report like this, that your website’s going to be a hundred percent accessible. Because as we’ve stated, these are automated tests, right? And automated tests only account for, you know, 30, 40 percent coverage of a test and everything else has to be evaluated manually.

Amber: You know, it just popped in my head. I was thinking about the WebAim Million, which is another similar study where they use the WAVE tool and they crawl the top million websites based on, they used to be Alexa ranking, which I don’t think is available anymore.

But, and they actually have a technology section. We can probably throw this up in our show notes later for people, if they want to look at it. And one of the things that I just like glancing at this WordPress is also sort of down at the, the bottom, but like, it’s just a little bit opposite because Drupal, for example, in theirs is better than WordPress.

Whereas this other report has Drupal way below WordPress. So, you know, how do you account for that?

Steve: It’s probably discrepancies in the tool itself, right? Like, I mean, WAVE is, you know, Amber just did a comparison of WAVE and the Accessibility Checker on our blog and WAVE is closed, right? WAVE’s closed. It’s not an open API. So we don’t have, we don’t know exactly how it’s evaluating. So if you run this same test on WAVE and you run it on Accessibility Checker and you run it on Lighthouse, right? The order is probably going to switch based on discrepancies between the different systems, the different scanners and which ones favor certain rules over, over the other.

So the moral of the story is, if you’re running a WordPress website, install Accessibility Checker. It’s free. Run a scan, see what’s going on.

Amber: And don’t suddenly think that you should switch to Wix.

Steve: No, not at all.

Amber: Is that our top takeaways? What’s your top takeaway, Chris?

Chris: I think I’m looking at this from more of a macro perspective, but if you pull back and like I was referencing earlier, you know, thinking about just market share and the size of the piece of the pie you’re having to manage and control the outcomes for, but I’m also thinking about the types of audiences that would gravitate towards a Wix or a Squarespace, versus a massive platform like WordPress, where there’s a lot more, I think, possibility in what you can build with that and the relative complexity of what’s out there. And so I’m just wondering to myself, if that’s yet another inherent advantage of these closed platforms, people are building simple five page websites.

I would argue is probably the majority of them or smaller eCommerce or maybe mid sized eCommerce. But you’re probably not going to see very many portals or really weird stuff, you know, that’s super unique and bespoke in, these types of platforms that are tied, you know, like an extension of a CMS, like in the HubSpot CMS hub or the Adobe Experience Manager, right?

I don’t know how complex that can really get. And I think with complexity, with size, with market share, as all of those increase, I think your likelihood to have potential accessibility problems introduced goes up too. And that is probably reflected in this measurement, which we’ve established is not the best measurement.

I know we’ve always been very upfront about, you know, automated testing tools are never the most ideal management, but having them, at least you’re measuring something. And then you can hopefully measure more deeply or manually test. As you go from there, but that’s the kind of stuff I’m thinking about.

I don’t know if I’ve drawn any conclusions as much as I think that this is yet another interesting data point and something to ponder over. But I don’t think it’s going to change my behavior or my outlook in any particular way. It’s just given me something to think about.

What about y’all? Any final thoughts?

Steve: Yeah. No, I think I think I’m in line with Chris. It doesn’t change a lot of my thinking. It was, it definitely is interesting where I started from with the initial read this, like, you know, I was like, at first I’m like, no, no way. Right. And now that I kind of read and I kind of do some testing, I’m like, oh, okay.

I could see that. And then like you evaluate. Okay. You know, the size of Wix user base based compared to WordPress and it being closed platform to an open platform, you know, like, yeah, I could see why those numbers would be higher but I think you’re right. The automated tools are not a perfect solution for everything.

They’re good tools to use to set the stage for your manual audit to kind of get a base standard of where you’re at. But I think that if they would have used a different tool the list would shift and so I don’t know, it’s an interesting take, I don’t think it really changes a whole lot.

I would not implore anybody listening to go install Wix because they think it’s going to be more accessible. Because you can achieve a very accessible website with WordPress and I mean, with a lot of platforms you can, but WordPress is open. There’s a lot that you can get in there.

And we didn’t even talk about that. I don’t want to go off , but like Wix being closed, can you really get that changed? I don’t have an H1, I don’t have an H1 on my website. Like I need to have an H1 for screen readers. Can you actually…

Amber: I mean, that one’s probably easy to fix. I bet you that was the website owner’s fault.

Steve: Yeah, maybe.

Amber: But some of those manual components, missing things. Or like ARIA that’s not available on a contact form or something like that. I, you know, no, you probably can’t fix that.

Chris: The response you get might be, we’ve put this in our bug issue tracker and we’ll evaluate it in three months and you could see a patch in a year.

Steve: It’s Is there a Wix community that can get in there and help you fix that issue? Like there is a WordPress community, right?

Amber: Yeah, I don’t know. So, so with my takeaway, you know, one of the things that I went and looked at is they do have a whole section on their website about accessibility. They say that they’re committed to, and all our features comply with the highest global standards, WCAG 2. 0. Now, first of all, 2. 2 is the current version, not 2. 0. That’s like 20 years old.

So I don’t know if that’s the highest global standard, if that’s what they’re targeting, but okay. Uh, they do say that they’re, you know, they recommend, and although this is interesting, you can actually find a Wix professional through their marketplace that specializes in accessibility.

So, you know, I would say if you have a website that needs to be accessible and it’s Wix, you probably do want to find somebody who knows what they’re talking about to audit it. And I wouldn’t just rely on the tool, the CMS itself.

Steve: Well, as we come to a close, I’m down to my cocoa sludge at the bottom of my cup.

Amber: So you’re about to make a second cup there, Steve?

Steve: This stuff is really good. I mean, I’m not kidding.

Amber: Does it beat out Diet Coke?

Steve: Oh, absolutely not. No.

Amber: Not enough caffeine.

Steve: Not enough. Not enough. I mean, yeah. We’ll get some more caffeine in here, maybe.

But this may be bad for the the old waistline, then keep drinking coke.

Amber: All right. Well, it’s been fun debating Wix as the best CMS. We are going to take a break for the holidays. So our next conversation episode will be out after the new year. We hope everyone has good holidays and a happy new year and we’ll see you online.

Chris: Thanks for listening to Accessibility Craft. If you enjoyed this episode, please subscribe in your podcast app to get notified when future episodes release. You can find Accessibility Craft on Apple Podcasts, Spotify, and more. And if building accessibility awareness is important to you, please consider rating Accessibility Craft five stars on Apple Podcasts.

Accessibility Craft is produced by Equalize Digital and hosted by Amber Hinds, Chris Hinds, and Steve Jones. Steve Jones composed our theme music. Learn how we help make thousands of WordPress websites more accessible at EqualizeDigital.com.