For one, I think Octillery is generally bad. It's too difficult to get out consistently unless you dedicate 4 of your deck spaces to it. It's also a liability on your bench with 2 retreat. And drawing up to 5 is pretty underwhelming, especially since it takes a turn to evolve into unlike Shaymin.
The biggest reason is that I was only focusing on two matchups, Yveltal and Greninja. Yveltal is going to get Garbodor out against me and I'm going to get Garbodor out against Greninja. Thus, Octillery is all but useless in those matchups. Shaymin helps my set up early game, has a good attack to conserve DCE and deal some chip damage to allow my Vespiquen to take easier KOs, and is just generally a better fit in the deck.
The real answer is that I didn't really think about it critically. I hadn't played the deck much before I went into the tournament, maybe 8-10 games at most the night before. I needed a cut for 3rd Blitzle and 3rd Zebstrika so I took one of each of Unown and Klefki out without considering just cutting 2 of one of them.
They both have their merits of course, Unown for the early draw and Klefki for making mega matchups slightly easier. If I were to play the deck again and I expected to play more than 1 Mega deck that's not Mega Rayquaza, I would go with 4 Klefki and 2 Unown. Otherwise, Unown is probably better and having 4 of it isn't a bad idea.
The one trick I did think about was being able to attach a Klefki to my Garbodor to activate Garbotoxin during my opponent's turn, while still giving me access to my abilities during my next turn. This could have come in handy against Volcanion to stop them from easily OHKOing you but still let you replenish your hand or place more Pokemon in the discard pile to take a big KO. This is where I like having the higher Klefki count but I didn't necessarily need all 4.
Long story short, try out all of the different combinations and see what you like best! I don't think there's necessarily a right answer. Just be deliberate with your Klefki drops, especially against a Fright Night Yveltal.
And, thanks for your kind words!