The problem with this myth is that it is simply untrue. It assumes that the Karaites and Sadducees interpret the verse "and they shall be for Totafot between your eyes" as referring to Rabbinic Phylacteries. However, in reality the Karaites and Sadducees never wore Tefillin at all, let alone between their eyes because this is simply not what the verse is talking about. One Rabbinite polemicist asked, 'How can you Karaites know how to make Tefillin without all the specifications laid down in the "Oral Law"?'. The answer is we can not because the "Oral Torah" made the whole thing up.

The phrase which allegedly commands the donning of Tefillin appears four times in the Torah (Ex 13,9; Ex 13,16; Dt 6,8-9; Dt 11,18). It should be noted that the difficult word "Totafot" which the Rabbis arbitrarily interpret to mean "Tefillin", actually means "Remembrance". This is clear from Ex 13,9 (one of the four "Tefillin" passages) which substitutes the word "Totafot" with the equivalent but more familiar "Zicharon" (Remembrance).

Upon closer examination it becomes clear that this phrase is a figure of speech and not a command at all. The brilliant Rabbanite commentator Rashbam (Rashi's grandson) was wise enough to realize the true meaning of this expression. Commenting on the verse "And it shall be for a sign upon your hand and a remembrance (Zicharon) between your eyes" he writes:

"'For a sign upon your hand' According to its plain meaning (Omek Peshuto), 'It shall be remembered always as if it had been written upon your hand' SIMILAR TO 'he put me as a seal upon your heart' (Cant 8,6). 'Between your eyes', like a piece of jewelry or gold chain which people put on the forehead for decoration" (Rashbam on Ex 13,9)

Rashi's grandson rightfully interprets the "Tefillin passage" as a metaphor which demands that we remember the Torah always and treasure it like a piece of fine jewelry. Rashbam and the Karaites realize that not everything in the Torah is to be taken literally as a command. The classic example of this is "And you shall circumcise the foreskin of your heart" (Dt 10,16). Obviously God is not commanding mass suicide but is rather commanding us to figuratively circumcise the foreskin of our hearts, i.e. remove our impurity and stubbornness and commit to his covenant with our hearts. While this metaphor was easy to understand it is less obvious what kind of metaphor lays behind "and it shall be for a sign upon your hand and a Remembrance between your eyes ". This question is clarified by several passages elsewhere in the Hebrew Bible:

"Listen my son to the teaching of your father and do not abandon the Torah of your mother; because it is a beautiful wreath for your head and a necklace upon your throat" (Prv 1:8-9)
"Do not let truth and righteousness leave you; tie them upon your throat, write them upon the tablet of your heart." (Prv 3:3)
"Keep my son the Mitzvot of your father and do not abandon the Torah of your mother; Tie them upon you heart always, don them upon your throat" (Prv 6:20-21)
In light of these verses the real meaning of the "Tefillin" passage becomes clear:
"And let these things which I command you today be upon your heart... and you shall tie them for a sign upon your hand and for a remembrance (Totafot) between your eyes and write them upon the doorposts of your houses and your gates" (Dt 6:8-9)

The Torah is to be like a fine bracelet or necklace which we are to wear proudly. In other words, the Torah is supposed to be precious to us and be remembered always. It is worth noting that of the four places in the Torah which use this expression 2 of them are telling us to remember the Torah (Deuteronomy 6:8-9; 11:18) while the other two are commanding us to remember the Exodus from Egypt (Exodus 13:9, 16). It should be noted that the Karaites also interpret the verse "And you shall write them on the doorposts of your houses and your gates" (Deuteronomy 5:9; 11:20) to be a metaphor equivalent to "write them upon the tablet of your heart." (Prv 3:3) and not as referring to the Rabbanite Mezuzah.

https://www.nehemiaswall.com/tefillin-phylacteries