Presented at the 2022 World Education Research Association, San Diego
This paper seeks to explain the mathematical and logical limits to policy modeling. In particular, it is suggested that mathematically vigorous, predictive models of social behavior and policy implementation are highly susceptible to unintended consequences, serious evaluation problems and policy failure. The fundamental logical and mathematical issues are explored through the modeling critique of Tomas Aquinas, mathematical chaos theory and the philosophical problems in bridging idealisms and observable reality. Further, economist Charles Goodhart’s “law” is consulted in order to demonstrate evaluation problems common in top-down reform efforts. Finally, a cyclical model of policy implementation is suggested for more effective reform.
Keywords: Policy Implementation, Idealism, Education Evaluation, Logical Limitations of Modeling
Top down education reform is usually derived from idealistic, axiomatic thought. The sequence in top down educational reforms is often as follows: 1. Choose ideal as axiomatic good. 2. Design policies in service of the ideal. 3. Design models for predicting the effectiveness of the policies under consideration. 4. Implement policies.
This sort of thinking is evident as far back as the Upanishads (Dasgupta 1969) but, in modern contexts, is more often pursued through the idealistic methods of Plato, St. Augustine and Marx. Owing to idealism’s inherent bias for perfect and unchanging forms over the imperfect and changeable world of experience this thinking has, from the beginning, been associated with mathematics (Russell, 1945). In Plato, this abstract, mathematical way of thinking is best illustrated in his metaphor of the cave. The “sleeping” people who merely grasp the knowledge of the senses are, according to his reasoning, awoken to truth only through grasping the perfect and eternal forms accessed through abstract and, particularly, mathematical reasoning. As DiRado (2015) says, “dialectical education involves shifting the noetic ‘gaze’ of the soul away from becoming toward being. The still asleep un-turned soul only considers the objects that it is engaged with as objects, and so fails to explore the being or natures that those objects exemplify. In contrast, the awakened and turned soul recognizes that the particular things it encounters only are intelligible insofar as they are instances of some nature—which is to say, insofar as they participate in Forms. If mathematical education has the power to prepare exceptional students for dialectic, it will be because it has the ability to reorient such students away from things to the constitutive characteristics that explain the intelligibility of those things.”
In a similar fashion, St. Augustine’s “City of God” recommends we turn away from the temporary and changeable things of observable reality and focus on the eternal fixity of mathematical certitude. Friedrich Engels in “Socialism: Utopian and Scientific” (1880) concurs with Plato and the saint and lays out the eternal, scientific and unchanging axioms of social organization as follows:
“The materialist conception of history starts from the proposition that the production of the means to support human life and, next to production, the exchange of things produced, is the basis of all social structure; that in every society that has appeared in history, the manner in which wealth is distributed and society divided into classes or orders is dependent upon what is produced, how it is produced, and how the products are exchanged. From this point of view, the final causes of all social changes and political revolutions are to be sought, not in men’s brains, not in men’s better insights into eternal truth and justice, but in changes in the modes of production and exchange.”
The reformer follows in the footsteps of Plato, St. Augustine and the Marxists when she attempts to impose top down reforms on education. The OECD’s “Universal Education Goals” function identically, on a structural level, with Plato’s forms. If a philosophical approach is used, it is possible to evaluate and critique these idealisms and reform attempts in at least four ways:
- Tomas Aquinas’ investigation of modeling and abstraction
- The unavoidable conflicts between material conditions and idealisms
- Goodhart’s Law
- The Confucian “Mun-jil-bin-bin” 文質彬彬 framework.
Through these approaches, it will be found that top-down, idealistic reforms suffer from a number of logical and practical shortcomings. It will further be found that the intended reforms thus imposed often suffer serious divergence from the intended, ideal state upon implementation. Finally, alternatives will be derived through the Mun-jil-bin-bin approach.
II Literature Review
Tomas Aquinas’s Investigation of Modeling and Abstraction
According to Aquinas’ argument (Kreeft, 2009), the reliability of modeling predictions are inversely correlated with the level of “realness” in the thing being modeled. That which is “perfectly unreal” can be predicted perfectly. That which is “absolutely real” can in no ways be predicted.
A pure abstraction, the mathematical equation Y=X+2 for example, can be modeled with absolute accuracy regardless of how large you make the value “X.” Aquinas explains this by claiming that these pure abstractions have a “realness” of zero.
In cases that Aquinas would characterize as “minimally real” – that is things that are very simple and very isolated – it is possible to make good but imperfect models of future outcomes. It is possible to visualize this by imagining a block of pure carbon floating in a vacuum. In such a controlled environment, a scientist could make accurate predictions of how quickly the carbon will sublimate, how prone it will be to cracking etc. However, the scientists will be unable to ascertain which atoms will sublimate at what time or to specify the molecules from which the cracks will originate. At this “minimally real” level of existence, therefore, modeling is capable of good but imperfect predictions.
We come next to Aquinas’ level of “complex reality.” A five-year-old girl provides here a convenient exemplar. Owing to the complexity of five-year-old girls and the complexity of the systems within which 5-year-old girls exist, any model of this girl’s life will be necessarily very limited. Where she will live, with whom she will marry, if she will marry, which social causes she will embrace and which she will reject, how she will support herself and how she will die – no model can realistically predict.
Aquinas proposes God as his “fundamental reality” and, consequently, the thing for which modeling predictions are inapplicable. Other philosophers have substituted logos, tao, the multiverse etc. for God. Such substitutions are, from the standpoint of the logic involved, unimportant.
The problem Aquinas describes is, fundamentally, a limitation of logic in general and mathematics in particular. In order to explain, Austrian mathematician Kurt Gödel’s (1931) “incompleteness theorem” is helpful.
- “Any consistent formal system F within which a certain amount of elementary arithmetic can be carried out is incomplete; i.e., there are statements of F which can neither be proved nor disproved in F.”
- “Assume F is a consistent formalized system which contains elementary arithmetic. Then F ⊬ Cons(F).”
In layman’s terms, the first theorem states that any consistent axiomatic system of mathematics will contain statements that cannot be proven within itself. The second theorem states that if there is a system allowing the proof of all statements, that system is inconsistent. Therefore, any such mathematical system that is consistent is false.
The UN and similar central authorities, when they propose “universal values” or “universal goals” are, in every sense, attempting to model Aquinas’ complex realities through Gödel’s incomplete axiomatic systems.
Compounding the issues pointed out by Aquinas and Gödel, there is also the problem of mathematical chaos (Stanford, 2008). A proper conception of mathematical chaos incorporates three characteristics:
- Extreme sensitivity to initial conditions.
- Deterministic progression
Mathematical chaos is thus extremely difficult to model over long periods. Weather is a representative case. The causes of weather are deterministic – high pressure zones, humidity levels, prevailing winds etc. – and they progress non-linearly. Weather is also extremely sensitive to changes in initial conditions. Because of the immense sensitivity to initial conditions, models predicting the weather become weaker and weaker as time passes. This is popularly imagined as the butterfly effect.
Even in conditions as tightly constrained – “minimally real” in Aquinas’ framework – as particle physics experiments, mathematical chaos cannot be avoided. This is a consequence of the Heisenberg Uncertainty Principle (Heisenberg, 1949). That is, the greater the certainty with which a particle’s location is known, the less its velocity can be known and vice versa. This is a fundamental feature of the mathematics, not a limitation in the measurement tools. Thus, even in the unrealistic eventuality that education reformers gather all the knowledge it is physically possible to possess about students/parents/shools/etc predictions must remain imperfect.
And it is unreasonable to expect reformers to have perfect or even good information. First, a classroom or school is an immensely chaotic system, bursting with uncontrolled variables. Second, any axiomatic model for education reform will need to be artificially simplified to be used by non-experts. Third, even if all important variables could be controlled in an educational setting, which they cannot, the educational setting is still infinitely more complicated than a physics lab. Paul Cilliers (2005) goes so far as to claim that even in ideal conditions, any model of such complex and chaotic systems will need to be as complex as the systems they model. Arendt explains in “On Violence” (1972):
“Events, by definition, are occurrences that interrupt routine processes and routine procedures; only in a world in which nothing of importance ever happens could the futurologists’ dream come true. Predictions of the future are never anything but projections of present automatic processes and procedures, that is, of occurrences that are likely to come to pass if men do not act and if nothing unexpected happens; every action, for better and worse, and every accident necessarily destroys the whole pattern in whose frame the prediction moves and where it finds its evidence.”
It follows that mathematically-minded education reformers, relying on social science “exactitude,” are prone to making precise models of things that do not exist. In the worst scenario these attempts to rigorously model reality cause the reformers to esteem their “objective” models over the practical but “subjective” results manifest in the field. Such eventualities, according to Frankfurt’s treatise “On Bullshit” (2016), tend to fall under the category of bullshit.
Conflicts Between Material Conditions and Idealisms
According to numerous studies, the formation of an idealism is rarely rational. The more typical process is a) emotion, b) judgment c) rationalization (Novella, 2012). Further, according to Nickerson (1998) and Martin (2015), when an idealism is threatened by external evidence, the most common response is to defend the idealism against external reality. This often done by portraying the threatening reality as taboo or evil (Dreger 2015).
In the 20th century, nothing exemplified this phenomenon more than totalitarianism. As shown by Garrido (2019), one of the distinguishing characteristics of totalitarian societies was a consistent preference for “objective” and mathematically precise ideologies over “subjective,” messy but manifest truths. Racial science and dialectical materialism were both, Garrido claims, attempts to eliminate “superstitions” and make observable reality conform to idealistic models.
St. Augustine’s “City of God” (1958) drawing on Neo-Platonic and early Christian thinkers, condemns on countless occasions the real condition of physically present Rome on the logic that it does not reach the perfection and eternal fixity of heaven. In a similar vein, Plato’s “Republic” (1992) declares the sensible world false on the logic it is not perfect and thus not in accordance with mathematical idealisms.
In the 21st century education field, the similarly idealized goals of the United Nations include eliminating poverty, universalizing comfortable work and supporting economic development, providing high quality education, combating global warming prevention and extending the reach of human rights ideology. As with earlier idealists, the United Nations struggles to apply its goals to field conditions. For example, the most successful reduction of poverty in human history was undertaken by the Chinese Communist Party in the latter part of the 20th century and early part of the 21st century. However, this poverty reduction almost could not have been worse for the environment and came about through a systematic rejection of human rights. (Li, 2003; Human Rights Watch, 2020)
Consider in particular the United Nations’ fourth sustainable development goal, quality education:
“By 2030, ensure that all learners acquire the knowledge and skills needed to promote sustainable development, including, among others, through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship and appreciation of cultural diversity and of culture’s contribution to sustainable development.”
Removed from Aquinas “pure abstraction” and brought into reality, the United Nation’s model of sustainable development falters. Consider the following examples:
1. How is one to respect gender equality and diversity when diverse cultures almost never respect gender equality? When Hmong tribesmen (Redfern, 2018) practice their diverse tradition of wife stealing, either the imperative for diversity or the imperative for gender equality must be sacrificed.
2. How is any non-violent society supposed to exist alongside diverse, militaristic societies? South Korea, Israel and Palestine, to name but three, would be subject to immediate existential dangers if they embraced non-violence.
3. How can citizenship ideas be extended to willfully isolated peoples in the Amazon Jungle or Indian Ocean? Even more fundamentally, how are we to offer citizenship education opportunities to aborigines whose immune systems cannot reliably survive contact with “global citizens?” (Endicott et al, 2003)
4. How can human rights coexist with the imperative to respect Cuban or North Korean sovereignty? If they cannot, does that give the United Nations permission to impose human rights on Cuba or North Korea in the violent, imperialist campaigns any swift imposition of human rights would surely require?
5. Can we realistically expect global warming prevention from diverse, national approaches? Wouldn’t it likely be more effective if a central authority has the power to enforce compliance?
Setting aside the otherworldly idealism of the United Nations’ Sustainable Development Goals, implementation research shows how, even if the logical gap between idealisms and the field are bridged, it is unsafe to assume the subjects of reform will accept “universal values.” Gottfried and Conchas (2016), Dahlin and Cronin (2010) and West (1994) found evidence that the people subject to top down reforms, rather than passively accepting the idealisms often sabotage said idealisms, especially when they were not consulted before implementation.
It is possible to ameliorate “universal values” style idealism through the concept of “emergence,” described by Mason (2008). In short, emergence theory proposes that additional layers of complexity create structures and phenomenon that cannot be inferred from the constituent parts. For example:
- The concept “Honda Civic” cannot be inferred from looking at 600 kg of steel, 300 kg of aluminum, 100 kg of glass and 100 kg of plastic.
- The concepts of 600 kg of steel, 300 kg of aluminum, 100 kg of glass and 100 kg of plastic can easily be derived from looking at a Honda Civic.
“Honda Civic,” existing at a higher plane of complexity, thus has several emergent properties we cannot deduce from lower level ideas like glass, steel etc. In a similar sense, there are many situations where the more complex and vastly more emergent reality of an educational system cannot be described by simple, idealistic “universal values.” According to Mason, emergent properties are very prominent in education policy and thus reformers are wise to avoid attempts to “deduce” ideal education practices from simple, constituent parts.
Economist Charles Goodhart’s law explains how indicators become less useful in response to public awareness.
“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” (Goodhart, 1984)
In layman’s terms, as soon as an indicator for evaluation becomes public, the subjects of evaluation will learn how to manipulate that indicator and it will cease to serve its purpose. This occurs because the intended purpose of the evaluation matters less than the indicator itself from the point of view of the subjects of evaluation. A representative example of this is the KSAT (Korean Student Achievement Test) and the phenomenon of cramming. (Yoon, 2014; Kornel, 2009)
In theory, the KSAT is supposed to measure students’ proficiency in math, language, history and other subjects. However, because the students being evaluated understand the indices used for evaluation they are strongly incentivized to ignore the intended purpose of the tests. Instead of learning the meaning of historical episodes or historical figures, for example, they memorize names and dates. Likewise It in English education, it is common to find students who scored highly on the English KSAT who are unable to form even simple sentences in conversation. According to studies (Kim et al, 1997; Cho, 2003), students studying for the KSAT in this way often experience terrible pain in order to achieve distorted test results and forget the “knowledge” gained soon after.
When the United Nations, the OECD or similar education authorities evaluate educational districts, schools or individual classes they face similar challenges. This is because central authorities use known indices to measure the performance of local school districts, schools and individual classes they incentivize the local administrators, principals and teachers to cheat in the same way. For example, if a central authority proclaims that, in order to achieve the universal ideal of gender equality, each school must achieve a female participation rate of at least 40% in STEM classes, the schools are prone to focus on the index (participation rate) rather than the intended effect (gender equality). In order to minimize the conflicts with teachers, students and parents such a policy would entail, the school might technically meet the criteria and avoid conflict by listing its cooking class as “edible chemistry” or the like.
Further, the more diverse the population being evaluated, the more prone to Goodhart effects the population will be (Signe, 2017). When a population has diverse opinions, the numbers of people who wish to undermine any given reform increases. In the STEM class example, support for implementation of universal values style reforms will likely be easier if the teachers, students and parents are all middle class or elite members of the dominant ethnic group (Dahmer, 2015). However, if the communities include ethnic minorities, working class people, poor people or immigrants, the chances of socially conservative elements introducing Goodhart effects increase dramatically. (Craig et al, 2017; Thompson, 2018; Zuccotti, 2018; Franklin, 2020) Therefore, especially in diverse schools or educational districts, cheating the index and deceiving the universal values-minded reformer is often easier.
According to Signe (2017) the fundamental problem with top down policy implementation research is dependence on natural experiments. This is possible to connect with Aquinas’ “levels of realness” concept. Furthermore, Pierson’s (2000, p. 252) “path dependence” concept illustrates how it is difficult to eliminate the gap between idealized policies pre-implementation and the reality of policy in the chaotic and imperfect field. For example, Pierson finds that people subject to policy reform most often fit their behavior to the oldest policy, even when that policy is manifestly broken, rather than the best policy.
Collier and Collier (1991) find results consistent with the principles of mathematical chaos theory. During the implementation of a reform, seemingly trivial factors often create large differences in outcome. They characterize policies as behaving like wild animals. Order of implementation, luck and timing, they note, seem to have large effects on policy outcomes at least as important as the content of the reform itself.
Similarly, Winter (2011 p. 24) finds that controlled policy experiments – that is experiments undertaken in controlled, laboratory conditions – are not applicable to field implementation. Winter further finds that no single theory can explain the subtleties of policy implementation and that reformers should implement policies slowly, in stages and with the intention to amend as they go. “Engineering approaches,” for this reason, are to be avoided (Signe 2017) (Head and Alford 2013). Head and Alford in particular echo the principles of Gödel, mathematical chaos theory, the uncertainty principle and Aquinas’ modeling framework.
IV Examples of Top-Down Policy Implementation Efforts at Korean Universities
For this section, interviews were conducted with university instructors and professors at a variety of Korean universities. Due to the political sensitivity and potential career damage involved with honestly discussing policy implementation, both the names of the participants and the names of the universities have been withheld. The top down reforms include the following goals:
Pursuant to these goals the following top-down polices appeared:
- There should be no more than 25 students in any single class.
- Professors should include “realistic experience” elements in all of their classes.
- Professors must include “workplace practice” in all courses.
In the field, the reforms fell victim to Goodhart effects, modeling constraints and idealism limitations in the following ways:
- There shall be no more than 25 students in any class. In one case, this reform was not implemented until shortly before the mid-term tests. Before implementation, students attended the Class Y for two hours on Tuesdays and one hour on Wednesdays. After implementation, schedules had to be suddenly rearranged. The students were officially divided between the original instructor and another instructor. In practice, however, all forty “divided” students simply attended the original professor’s class on Tuesday and then attended a second professor’s class on Wednesday. This disrupted syllabuses for both instructors, required the second professor to improvise content without preparation time, did nothing to reduce the size of classes the students experienced and wasted administrative resources.
This illustrates why it is dangerous to rely too heavily on “objective” measurements. By focusing on easily verified, objective criteria – class size on an attendance sheet – the reformers invited Goodhart effects and marginalized subjective factors like the late implementation of the reform, the continuity and flow of classes and the instructors’ individual expertise and preparation.
- Professors should include “realistic experiences.” In order to ensure easy and objective evaluation, the reformers evaluated compliance almost entirely through paperwork submissions. For instructors already including “realistic experiences” in their classes, the reform amounted to greater administrative burdens and little more. For instructors who did not wish to include “realistic experiences,” adjustments in their paperwork unaccompanied by changes in class structure sufficed.
This problem manifests Goodhart effects. Instructors, recognizing the indices under evaluation, ignored the intended effect of the reforms, manipulated the indices and carried on as before.
- Professors must include “workplace practice.” Because easy and objective evaluation was a priority, the authorities required electronic copies of practice videos and paper copies of students’ plans. The practice videos and plans evolved into strange, choreographed “work dances” where students would cycle through rehearsed, highly artificial rituals reminiscent of children’s television.
This problem exemplifies both Aquinas’ modeling problems and Mason’s emergence concept. Due to the complexity of university departments the number of uncontrolled variables is immense. The “work dances” are an example of uncontrolled variables manifesting in unforeseen behavior.
- This policy was, according to the instructors interviewed, most successful. Professors and students agreed on a formalized schedule. The instructor and student then worked face to face to improve the student’s essays. The students were free to bring essay assignments from any subject – sometimes even bringing non-class materials like personal stories or resume drafts. The instructors were, likewise, free to advise the students in whatever way they saw fit.
The success of this program, according to Aquinas, Gödel, Goodhart etc., is due to the following factors: a) Wide agreement between staff and administrators that one on one interaction is positive for students, as explained by Signe (2017). b) The degree of freedom given to both students and instructors for implementing the reform. c) The very generalized and flexible nature of the administrators’ “idealisms.”
- In practice the contextual factors mentioned by Head and Alford (2013) manifested in destructive ways. Some instructors felt that their reflections were not being read and were thus purely a paperwork burden. Other instructors worried that their reflections would be read and later used as weapons against them. In both cases, instructors largely filled their reflections with “bland nothingness” – bland nothingness satisfying the evaluation standards, being easy to produce and protecting the instructor from attack.
- In practice, many staff found this requirement onerous. They often actively sabotaged this reform. Word of mouth complaints about university administrators almost certainly outweighed any marketing efforts made by the unwilling amateur advertisers. This manifests the inherent difficulties in translating an ideal into practice and the irreducible differences between models and field conditions as mentioned in Arendt (1972).
In summary, the problems considered above stem from the fundamental, logical limitations of model building and top-down reform.
V Conclusion and Suggestions – Mun-jil-bin-bin 文質彬彬
The Mun-jil-bin-bin concept derives from the Analects of Confucius. According to Confucius’ reasoning mun (文) – meaning writing or word – can be understood as theory. Jil “質” – meaning quality – can be understood as the qualitative experience of the world. Bin “彬” means to shine or glimmer and is repeated for emphasis. Confucius uses this formulation to describe the “gunja” 君子 – meaning “man of virtue.”
Confucius imagines the men of virtue embracing Mun-jil-bin-bin because the path to wisdom, in his view, entails beginning with mun – theory – and uniting it with jil – the qualitative world – until both shine brightly. A man of virtue is to start with a theoretical understanding of the world so that he can approach it intelligently. This is by necessity a type of idealism. However, in order to avoid the numerous pitfalls of idealism, the man of virtue is to apply his theories to the qualitative world of experience. With this experience, the man of virtue is to create a second theory with which to better explain the world. By means of the second theory, he is again to launch into the world of qualitative experience. This process can and should be repeated indefinitely with the goal of approaching closer to, but never arriving at, ultimate truth.
Theory -> qualitative experience -> revised 2nd generation theory -> qualitative experience on the 2nd level -> 3rd generation theory -> qualitative experience on the 3rd level -> 4th generation theory -> contd.
Due to the problems specified by Aquinas and others, the perfect ideal, model or theory is not possible. However, with the Mun-jil-bin-bin approach, reformers can replace the quixotic pursuit of perfection with the reasonable goal of “better.” Further, by emphasizing contingent and imperfect idealisms rather than eternal truths, the Mun-jil-bin-bin approach necessitates a humble and imperfect view of “truth.” This has many advantages.
First, if the truth is by nature ambiguous it is difficult to maintain fundamentalist or totalitarian ideologies. The “scientific prophecies” of race theory and Marx – things which Arendt (1948) blamed for both Nazism and Bolshevikism – are far less credible in a Mun-jil-bin-bin framework. Second, if researchers can approach closer to truth through the Mun-jil-bin-bin or similar methods, both nihilism and scientism can be discredited (Garrido 2019) (James, 1910) (Nietszche 1896). Third, Mun-jil-bin-bin and similar frameworks allow the integration of diverse viewpoints and traditions with progressive reforms and scientific advances. For example, we can see the dangerously idealistic systems of late Joseon replaced with philosopher Jeong Yak-yong and King Young-jo’s practical learning system(Geum, 2001). Likewise, by replacing the otherworldly authoritarianism of Plato with the inductive and relatively flexible systems of Aristotle, Spinoza and Newton (Russell 1945) the Enlightenment emerged from feudalism while the Catholic Church was able to replace the otherworldliness of St. Augustine with the moderating influence of Aquinas (Kreeft 2009).
Incorporating Mun-jil-bin-bin or similar approaches is difficult for the OECD, United Nations or Korean Ministry of Education because these authorities all suggest the existence of “universal values.” Because of the fundamental idealism at the heart of “universal values,” however those values end up being interpreted, the modeling problems mentioned in this paper are endemic.
In particular, this paper recommends the following measures to approximate the Mun-jil-bin-bin cycle and minimize the logical and mathematical problems associated with top-down reform efforts:
- Educational reforms should include an expiration date. If policies and reforms are temporary, it is systematically and administratively easier to integrate adjustments with each cycle of revision. Reformers and the subjects of reform are also, with a policy of temporary policies, strongly incentivized to re-evaluate policy effectiveness on a regular basis, eliminate outdated policies and streamline administration.
- Educational central authorities should not too tightly control the implementation of their reforms. Instead of specifying the means of reform, it is better to focus on desired outcomes. For example, it is better for an educational authority to declare “schools are expected to boost female participation in STEM-related classes and report on their methods” than it is for the authority to specify “each school must include 50% female students in the robotics club.” This makes it easier for local authorities to bridge the gap between idealized outcomes and field conditions.
- In order to reduce Goodhart effects, relying entirely or mostly on objective evaluation strategies is less advisable than increasing subjective and difficult-to-predict evaluation strategies. For example, if evaluators arrive unexpectedly at a school or classroom, they are more likely to observe the levels of implementation of reforms on a “normal day” rather than a rehearsed or stage managed “implementation” likely to occur if the subjects of evaluation are given objective standards and a definite date. Further, on tests like the KSAT, it is advisable to substitute objective questions such as “what year did Dwight Eisenhower become president of the United States” with subjective questions such as “explain the significance of Dwight Eisenhower’s reforms to the United States.” These subjective questions are more difficult to evaluate, but they incentivize the learner to study the meaning of the subject matter rather than memorizing incidental trivia for the test.
Indian Idealism, Surendranath Dasgupta. 1969. Cambridge University Press
Perception and Judgment in Plato’s Theaetetus, Paul DiRado. 2015. Uknowledge
City of God, Saint Augustine of Hippo. 1958. Image
Socialism: Utopian and Scientific, Frederick Engels (Edward Aveling translation). 1880. Swan Sonnenshein & Co.
The Modern Scholar: The Philosophy of Tomas Aquinas, Peter Kreeft. 2009. Recorded Books LLC
The Origins of Totalitarianism, Hannah Arendt. 1948. Houghton Mifflin Harcourt Publishing Company
Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I (On Formally Undecidable Propositions of Principia Mathematica and Related Systems I), Kurt Gödel. 1931. Monatshefte fur Mathematik
Chaos, Stanford Encyclopedia of Philosophy. 2008.
The Physical Principles of the Quantum Theory, Werner Heisenberg. 1949. Dover Publications
On Violence, Hannah Arendt. 1972. A Harvest Book, Harcourt Inc.
On Bullshit, Harry Frankfurt. 2005. Princeton University Press
Your Deceptive Mind: A Scientific Guide to Critical Thinking, Steven Novella. 2012. The Teaching Company
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises, Raymond S. Nickerson. 1998. Review of General Psychology
How Ideology Has Hindered Sociological Insight, Chris C. Martin. 2016. The American Sociologist Volume 47 Issue 1
Galileo’s Middle Finger, Alice Dreger. 2015. Penguin Books
Republic, Plato. 1992. Hackett Publishing Company, Inc.
Energy and Environmental Problems behind China’s High Economic Growth – A Comprehensive Study of Medium – and Long-term Problems, Measures and International Cooperation, Li ZhiDong. 2003. IEEJ: March 2003
China, Events 2020, Human Rights Watch. 2020. Retrieved from https://www.hrw.org/world-report/2021/country-chapters/china-and-tibet
‘I’m too tired to cry’: The Laoitioan girls who are kidnapped to become child brides, Corinne Redfern. 2018. The Lilly
The Genetic Origins of the Andaman Islanders, Phillip Endicott, M. Thomas, P. Gilbert, Chris Stringer, Carles Lalueza-Fox, Eske Willerslev, Anders J. Hansen, Alan Cooper. 2003. The American Journal of Human Genetics
Totalitarianism, Democracy and Education Through Hannah Arendt’s Superfluity Framework, Ben Garrido. 2019. World Education Research Association, Tokyo Conference
A History of Western Philosophy, Bertrand Russell. 1945. Touchstone
Sustainable Development Goals. United Nations. Retrieved from https://www.un.org/sustainabledevelopment/sustainable-development-goals/
Sustainable Development Goals: Goal 4: Quality Education. United Nations. Retrieved from https://www.un.org/sustainabledevelopment/education/
When school policies backfire: how well-intended measures can harm our most vulnerable students, Michael A. Gottfried, Gilberto Q Conchas. 2016. Harvard Education Press
Achievement Gaps and the Proficiency Trap, Michael Dahlin, John Cronin. 2010. Northwest Evaluation Association
A Desegregation Tool That Backfired: Magnet Schools and Classroom Segregation, Kimberly C. West. June 1994. The Yale Law Journal
Problems of Monetary Management: The UK Experience, Charles Goodhart. 1984. Palgrave, London
Cramming Feels Effective Because of a “Brain Mechanism,” Yoon Tae-hee. October 13, 2014 Edition. Now News벼락치기가 효과 있다고 느껴지는 “뇌의 메커니즘,” 윤태희. 2014 10 13. 나우뉴스
Optimizing learning using flashcards: Spacing is more effective than cramming, Nate Kornell. Applied Cognitive Psychology Volume 23, Issue 9
The Harsh Truth About Progressive Cities, David Dahmer. 2015. Madison 365. Retrieved from https://madison365.com/what-no-one-wants-to-talk-about-race-and-progressive-cities/
Hispanic Population Growth Engenders Conservative Shift Among Non-Hispanic Racial Minorities, Maureen A. Craig and Jennifer A. Richeson, 2017. Social Psychological and Personality Science 1-10
Reasons Why Ethnic Minorities Have Higher Levels of Religiosity, Karl Thompson, 2018. ReviseSociology
Does ethnic concentration influence gender role views?: a study across ethnic groups in England and Wales, Carolina Viviana Zuccotti. 2018. Migration Policy Center
In defense of working class conservatism, Peter Franklin. 2020. UnHerd.
Students’ Stress in China, Japan and Korea: A Transcultural Study, Kwang-lie Kim, Hotaek Won, Xiehe Liu, Peiyi Liu, Kenji Kitanishi. 1997. International Journal of Social Psychiatry
Suicidal ideation, acculturative stress and perceived social support among Korean adolescents, Cho, Yong-beom. 2003. Dissertation Abstracts International: Section B: The Sciences and Engineering, 63 (8-13), 3907
Policy Implementation – A synthesis of the Study of Policy Implementation and the Causes of Policy Failure, Landry Signe. 2017. Research Papers & Policy Papers 1703, Policy Center for the New South
Increasing Returns, Path Dependence, and the Study of Politics, Paul Pierson. 2000. The American Political Science Review
Shaping the Political Arena: Critical Junctures, the Labor Movement, and Regime Dynamics in Latin America, Ruth Berins Collier, David Collier. Princeton University Press
Implementation, Soren C. Winter. 2011. International Encyclopedia of Political Science
Wicked Problems: Implications for Public Policy and Management, Brian W. Head, John Alford. 2013. Administration & Society Volume 47, Issue 6
Undistinguished School Experiences of Distinguished Persons, Henry Pang. 1968. Adolescence Volume 3 Issue 11
An Investigation of Dasan’s Practical Learning, Geum Jeong-tae. 2001. Sohaksa. 다산의 실학 탐구, 금정태. 2001. 소학사
The Nazi Primer, Adolf Hitler. 1938. Harper & Bros.
Action Research and Community Problem Solving: environmental education in an inner-city, Arjen Wals. 1994. Education Action Research . Volume 2. No. 2.
Action Research and Minority Problems, KurtLewin. 1946. Journal of Social Issues 2.
A Biological View of Our Foreign Policy, P. Charles Michel. 1896. The Saturday Review
Pragmatism and Other Essays, William James. 1910. Washington Square Press
Thus Spake Zarathustra, Friedrich Nietzsche (Alexander Tille translation). 1896. MacMillan
 For he (Plato) saw that the causes of things were sought for by them,—which causes he believed to be ultimately reducible to nothing else than the will of the one true and supreme God,—and on this account he thought they could only be comprehended by a purified mind; and therefore that all diligence ought to be given to the purification of the life by good morals, in order that the mind, delivered from the depressing weight of lusts, might raise itself upward by its native vigor to eternal things, and might, with purified understanding, contemplate that nature which is incorporeal and unchangeable light, where live the causes of all created natures. (Augustine, Ch 3.)
 Engels believes his “materialism” is not idealism. Scholars might quibble with Engel’s belief on technical grounds but dialectical materialism, as proposed by Engels and Marx is at least similar in function with any other axiomatic system of idealism.
 It is for this reason that, according to Arendt (1948), governments attempting to impose scientific predictions on society – the progress of genetics in Nazi racial theory, the inevitable path of dialectical materialism in the Soviet Union and the Khmer Rouge among others – they must work to limit the complexity of their citizens. By severely crippling individual freedoms and destroying access to information, the scientific model builders in such societies make their citizens more like cubes of carbon floating in a vacuum and, as such, make the moral and scientific models more predictive.
 In order to understand why the distinction between God, the tao and the multiverse makes no difference to the modeling problem, consider the case of a mundane object – fried tofu for example. If my question is “how much fried tofu will exist as a result of X’s activity?” it is equally impossible to answer whether X is God, the tao or the multiverse.
 In the best case scenario, mathematical approaches to social science can be used as a way to better observe and understand manifest phenomenon. “Munjilbinbin” is a way to do this and will be considered later in this paper.
 This process superficially resembles the “action research” as proposed by Wals (1994) and Lewin (1946). However, in contrast with these approaches, no democratic or community involvement is required in the Mun-jil-bin-bin process. This makes Mun-jil-bin-bin both more resistant to bottom up reform problems and more practical to individual reformers.