The Blacklist twist prediction

Spoiler warning: this post reveals plot details of the season 5 finale of The Blacklist.

In last night’s episode it was revealed that the bones in the duffle bag belongs to Raymond Reddington, which implies that the man we’ve all come to know throughout the last five years is not who he says he is. This twist was shocking to me. I have made a prediction since the end of last season that the bones must surely belong to Katarina Rostova, and that the secret that Red has been keeping from Liz is that he killed Katerina in order to pacify her enemies, so that they would not find their way to Liz as a means to get to Katarina. This turns out not to be the case.

This begs the question: who is the man we know as Raymond Reddington, who is now known to be a fraud? My guess is that ‘Raymond Reddington’ is in fact Katarina Rostova, after a gender reassignment to become a man.

While somewhat far fetched, this theory is seemingly consistent with the events so far. We first return to the 4th season episode ‘Dr. Adrian Shaw: Conclusion’, where Red whispered something to Alexander Kirk’s (Constantin Rostov) ear which somehow absolutely convinced him that he could not possibly be Liz’s biological father. Presumably, the only person who would know the truth is Raymond Reddington, the actual biological father of Liz… but it seems strange that Alexander Kirk would take his word so easily, with no further proof. Indeed, the only person who would really know the true parentage of Liz would be her mother, and so it makes sense that when ‘Red’ revealed himself to actually be a transformed Katarina, Constantin would instantly be convinced of the truth.

Next, given that we now know that ‘Red’ is not actually Raymond Reddington, one must interpret the episodes Cape May and all the other flashback episodes involving Katarina differently. In all of these episodes, ‘Red’ makes it absolutely certain that ‘Katarina’ is dead. However, now that we know that the bones do not belong to Katarina and even that ‘Red’ is not Raymond Reddington, a highly plausible explanation then is that ‘Red’ knows Katarina is ‘dead’, because he no longer has that identity and that he is now living as ‘Raymond Reddington’.

Finally, the episode ‘Requiem’ takes on a whole new meaning when viewed through the lens of our post-season 5 finale knowledge. The cryptic way that ‘Red’ spoke to Mr. Kaplan in their ‘first’ meeting now seem to carry the implication that he knows her. There is essentially no way that he would know if he was an actual random impostor… the most plausible explanation is that he is Katarina. Further, in this episode it was revealed that it was Katarina who rescued Liz from the burning house, and recall that in an early episode (either in season one or two) it was revealed that ‘Red’ had burn scars all over his body, presumably from having rescued Elizabeth from the fire. This strongly implies that ‘Red’ and Katarina are the same person.

In an earlier post  I made the prediction that Elizabeth Keen did not in fact die in child birth, and it seemed correct. Of course, that was much less of a long shot compared to the prediction above. Let’s wait until next season to see whether my guess pans out.

A discussion on probability

In a relatively recent discussion, the topic of applied probability has entered the fray, so I am taking the time to discuss this issue. The discussion is centred around a reflection of the debacle that was the 2016 US presidential election. Vitriolic politics aside, one notes that objectively, the prediction for the outcome of the election was an abject failure. Nearly all news outlets and pundits gave the Democrats a better chance for winning, and of course, they were all ‘wrong’.

There is a lot of contention with use of the word ‘wrong’ in this context. After all, just because there is a 95% chance that something will happen doesn’t mean it’s guaranteed to happen. However, I am claiming that assigning a ‘probability’ to something like a presidential election is fundamentally misapplying the principles of probability to a real world phenomenon.

Probability has been an absolutely invaluable tool in the sciences and economics, because it really helps us understand phenomena at all scales and how to interpret problems where we are only given partial information. But the underlying principle of probability is that events are similar, in the following sense: each time you flip a coin there is no reason to believe that it’s different from any other time you flip a coin, so the behaviour of the outcome should be similar. Therefore, one can concretely say that if you flip a coin a large number of times, that you will expect to see a certain proportion of coins being heads, and that this is the ‘probability’ that after a single coin flip you will get heads. This can be applied in any setting where similar, indistinguishable events happen in large numbers. Even for coin flips we know that not each coin flip is exactly the same (for example, maybe the temperature of the room is slightly different, the fatigue level of your hand is slightly different, etc) but we understand that these extra factors should be negligible. This allows us to apply, for example, probabilistic models on consumer behaviour even though we know that each person is different.

However, this basic principle is violated in models trying to predict large, singular events like elections. There is no reason to expect (the 2016 election of all elections) that a particular presidential election has any parallel in history or in the future. The candidates are completely different, the general political atmosphere is completely different, demographics are changing, etc. Thus the very notion of ‘probability’ is bunk: you can’t repeat it to check if your guess is correct, and models which cannot be repeated usually aren’t valid in science. Thus, one should step away from the pseudo-mathematical (mis)-application of probability in this subject and instead focus on whether one can actually predict the outcome of an election. On that note, this guy seems to have the right idea (see this article).

 

The Professional Effect

In a recent discussion with a friend, I talked about why the ability of polls and  markets to predict political outcomes seem to have taken a beating as of late. I described an effect whereas the more faith people place into a certain predictive tool and gain more knowledge of it, the less reliable it becomes. I vaguely remember hearing about it somewhere but when pressed for an academic source, I came up short. I couldn’t even give a name for it, which leads to the disturbing possibility that I simply made it up.

However, weeks later, I found a name for it in a rather unorthodox source: this video on the video game Hearthstone. At around the 11 minute mark, Trump talked about something he named the “professional mage effect”, which I shortened to simply the professional effect, which I believe is one of the main causes in the crumbling of the accuracy of polls and other statistical tools as of late. Trump himself summarized the effect fairly succinctly in the video, but for those of you who don’t know the language of Hearthstone, I will give you an explanation.

In Hearthstone there are nine playable classes. Each class has a distinct set of accessible cards and hero abilities. In the current incarnation of the game, the mage class was given several new cards that gave it a significant advantage over other classes. This meant that every time it was possible to select a mage, people tend to select mage; the difference in win rate between mage and the other 8 classes was staggering. As a consequence, Blizzard decided to introduce some changes to try to balance the game, most notably removing several problematic cards for mages. However, after a month of the changes, it seems that mage still maintained a sizeable lead. Blizzard explained this with the ‘professional mage effect’, where strong players who tend to read the statistics instead of choosing at will or from word of mouth, tend to still prefer mage because its previous high win rate and its current average win rate still gives an above average win rate; and since these players tend to be the strongest, their superior skill allows them to win more, which is conflated with the class being strong.

It appears that a similar phenomenon is at play when it comes to high frequency trading. Whenever too many algorithms use similar methods to choose stocks, a slight advantage is exaggerated and causing many buyers to buy into it, significantly overvaluing the stock. This then creates a lot of uncertainty in the valuation of stocks and makes the market rife for bubbles.

This seems to give a rigorous justification that a diversity of ideas is quantitat

新三国剧《军师联盟》

作为一个铁三国迷，我非常期待吴秀波主演的新三国剧《军师联盟》。看了片花以后，觉得潜力非常大，有可能有让历史焕然一新的作用。

在大众心目中的三国故事，三国演义，其实是元末明初人士罗贯中所写。演义里面的人物正邪鲜明，明显是一个捧蜀汉贬曹魏的著作。最近国内两大三国演义电视剧都没有摆脱这个以刘备为英雄，曹操为枭雄的基本故事结构。最起码《新三国》里面的司马懿戏份还算不少，但仍然是一个负面角色。

然而看完《军师联盟》片花，可以看得出一些与小说不同的说法。其一，故事的主角是司马懿，也就是早就最后三国归晋的政治家和军事家。司马懿是在董卓乱朝，诸侯割据之后才入仕，所以按道理不算是什么汉臣，也就没有跟曹操那样的道德枷锁。再者，司马懿此人的故事很复杂。他辛辛苦苦伺候了三朝大魏郡主（包括曹操在内），其实可以说的上是忠臣。但是到老却无奈曹魏重用曹姓人的秉性，被排挤，甚至有功高震主而有可能招来杀身之祸。最后他只能反击，独揽朝野，造就了晋代魏的局面。虽然他是中国一个朝代的“太祖”，却永远在故事里被描述成曹操的属下或诸葛亮的劲敌，永远不是故事核心。所以这出戏有潜能能把一个杰出历史人物的故事讲好。

另外一个剧情可能没有多少人留意。在三国历史上被黑的最严重的人非蜀后主莫属。“阿斗”已经成为笨蛋庸才的代言词，而每一个跟三国有关的文艺作品都会把刘禅写的一文不值。三国演义为了抬高蜀相诸葛亮的形象，更把后主写的跟猪一样。但是历史上的后主可能不是什么明君圣贤，但是最起码他以一个外来人（刘备是从外地征服蜀地的），年轻人（后主上位时才十七岁）的状况下成功的团结了蜀地原来的官僚和士绅与刘备从荆州带来的部下和军队。从这看来刘禅绝对不是什么笨蛋，而是拥有一定政治能力的。在片花了，后主大怒把走着从案上一扫而下，悲叹“十万大军！这已经第六次了！好一个‘忠臣良相’！”他说的第六次肯定是说诸葛亮六出祁山，而他的悲叹很明显：孔明率领大军北伐，对经济民生造成了极大的负担，也是最后导致蜀国国弱民衰的原因。这种叙述诸葛亮的写法好像从来没在戏里看过，一定很新鲜，很让人值得思考。

希望能尽快看到！

An ethical question with a concrete answer

Ethics questions often give a scenario (often unrealistic and missing key details) where you have to choose between a set of difficult options. For example, it could be “an old lady who has no family or a young man with a promising career and loved by a lot of people are both admitted to the ER. Who would you prioritize on treating if their probability of survival is the same?”

However, sometimes the answer is quite a bit more straightforward. The question I saw is the following: Suppose that an intelligent machine has the ability to predict, with 99.99% accuracy, whether someone will commit murder. Would it be permissible to arrest people based on the predictions of the machine?

There seems to be no ‘correct’ way to answer this question, but that’s because the person who asked the question doesn’t understand statistics. There is a phenomenon in statistics called the false positive paradox, where even a very accurate test will produce many more false positives than actual positives. This is relevant when the actual probability of a true positive is very low.

Here is an example. You are a unique person, one of say 7 billion. Suppose that there is a machine that can identify someone with 99.9999% accuracy. A person gets scanned by the machine, and the machine says it’s you. What is the probability that the machine is right?

There are two possibilities for the machine to give that reading. Either the person scanned by the machine actually is you and the machine is accurate, or the person scanned by the machine is not you and the machine malfunctioned. The probability that the machine is accurate is 99.9999%, and the probability that the person scanned is you is one in 7 billion. The other possibility is that the machine is wrong and the person being scanned is not you. The probability that the machine is wrong is 0.0001%, and the probability that the person is not you is 6,999,999,999 out of 7 billion. Thus, the probability that the person actually is you, given that the machine says it scanned you, is given by the equation

Evaluating, the probability that the machine is right is less than 0.015%. The paradox is caused by the fact that the machine’s accuracy is very poor compared to the astronomically unlikely phenomenon that you would be picked out of 7 billion people.

Now back to the question. Whether it is ethical or not (that is, whether the machine produces a desirable result an acceptable proportion of the time) will depend on the actual murder rate of a place. The global average is currently 6.2 people out of every 100,000, or 31 out of every half a million. Assuming the machine has accuracy 99.99%, the probability that a person identified by the machine as a murderer is actually a murderer is given by

or roughly 38.3%. Therefore, the machine is right way less than half the time, far below what can be considered reasonable. Therefore this question has a concrete answer and is not really debatable.

Some thoughts after reading an article on Peter Scholze

Recently the following article appeared on my Facebook feed: https://www.quantamagazine.org/20160628-peter-scholze-arithmetic-geometry-profile/. Aside from the usual bland mix of singing praises of an archetypal ‘genius’, the article does contain some genuine insights. The most striking of which is Scholze’s description of him learning the proof of Fermat’s Last Theorem by Sir Andrew Wiles: that he “worked backward, figuring out what he needed to learn to make sense of the proof”. Later he also said things like “I never really learned the basic things like linear algebra, actually – I only assimilated it through learning some other stuff.”

If you have experiences learning mathematics at the senior undergraduate level or post-graduate level, you will likely find that these experiences are orthogonal to your own. We spend an inordinate amount of time learning the ‘basics’ for various things, which very much includes linear algebra for example, in order to do research… or so we are told. If you have passed the part of your career where you do more courses than self-learning, then you have likely reached the epiphany that usually it’s not efficient to learn everything there is to know on a subject before actually doing work on the subject.

Some of you have had advisors telling you things like “read these five books (each 300+ pages) before you attempt any research work in the area”. Sometimes this advice comes from highly proficient researchers, which seems odd: if the above ethos is ubiquitous among researchers, why tell your students to do something totally different and entirely more dreadful? I am not sure what the right answer is, but probably part of the reason is a misguided attempt to make research ‘easier’ for students. Perhaps many advisors recall the struggle of trying to understand ‘simple’ phenomena that they encountered in their research careers, that if someone had just told them to read a book or if they were better prepared, would have been trivial to overcome. Perhaps they wish to save their students some time by telling them the shortcut. However, the struggle to understand phenomena on your own is part of what makes research rewarding, and more importantly, it is critical in forging a mind suited to making discoveries.

Of course, I am but a pebble to the avalanche that is Peter Scholze, so my advice may not be worth much. Nevertheless, I feel like I should say this to all prospective and current graduate students: be bold, and give every difficult paper in your field a read. Don’t be intimidated by them. If you don’t understand something, google it until you find what you need to learn the language of the subject. Don’t feel like you need to understand all of Harthshorne before you can read any research papers related to algebraic geometry. Your future self will thank you for this.

Manjul Bhargava’s advice to mathematics graduate students

Last night, I had the honour of attending a panel discussion featuring eminent mathematician Manjul Bhargava. During the panel, the moderator, Professor Kumar Murty asked the very productive Fields Medalist to give some advice to graduate students in the audience who may be struggling with their research. Professor Bhargava’s response, paraphrased, is essentially the following:  always work on several problems at a time (at least three), of varying difficulty. There should be a problem which is quite difficult and if you make any progress on it it will be a major breakthrough; there should be one of moderate difficulty, and there should be an ‘easy’ problem that you know you can make progress on eventually. Further, never think too much about a problem at a time and instead rotate between the problems to change your mindset. Sometimes when you approach a problem with a fresh perspective you will gain some insight that would’ve been impossible if you stared at the same problem continuously, since you are subconsciously trying to apply the same techniques.

I thought Professor Bhargava’s advice was very helpful to the graduate students in the audience. It is something I started doing a few years ago, but it wasn’t something that I was aware of consciously. Hopefully heeding this advice will be helpful to your work.

Elizabeth Keen is not dead

More so than that, it appears that my observations watching the episode where ‘died’ were basically right on the money. Feels good to be right!

Why Elizabeth Keen is (obviously) not dead

I just watched the latest episode of The Blacklist. MAJOR SPOILERS ARE AHEAD, THIS IS YOUR WARNING.

So in the latest episode, “Mr. Solomon – Conclusion”, there is a cataclysmic plot twist in which Elizabeth Keen, whose mysterious familial background is the driving force of the show, seemingly perished after a medical complication. The shock value of this development cannot be underestimated… however, I claim that she is (obviously) not dead. Here are the clues:

1) When Keen was assessed by the first doctor at the hospital, her diagnosis was fine. There are no indications at that time that she has suffered any serious trauma.

2) When Mr. Kaplan went to pick her up from the hospital, Keen was talking angrily about how she’s in this situation because of Reddington. Mr. Kaplan looked sympathetic. Most likely they planned what to do right after.

3) Notice that Keen started developing medical complications only after being treated at the nightclub. During this time Mr. Kaplan, Keen, and Nick had time to converse in private; Reddington (and therefore the audience) is not aware of what happened. After this conversation, all of a sudden Keen’s medical condition started deteriorating, and Nick asked for additional medical equipment to be brought. Who took care of this? Mr. Kaplan.

4) Mr. Kaplan very squarely blamed Red for Keen’s predicament. It is not at all surprising if she would secretly plan to fake Keen’s death just so that Reddington could not be in Keen’s life any longer.

5) After Keen has been pronounced dead, Mr. Kaplan, who earlier in the episode showed that she obviously cares for Keen, displayed no distress. Reddington almost fell over and Samar balled her eyes out, even Nick was extremely distraught. So why didn’t Mr. Kaplan react at all? Probably because she knows Liz is not really dead. Moreover, she again murmured something to Nick while Dembe was helping Red into the car. Clearly there is something astray here.

The obvious need to ‘kill’ Liz is because Megan Boone is pregnant for real and probably has to go give birth for real, and take some mat leave on top of that. This is a pragmatic solution to this very gnarly logistical issue. Do not be surprised if Keen pops up all of a sudden on episode 20 or earlier.

 

Mathematicians have always been willing to accept new ideas

In a recent publication (see here) of a popular internet comic strip that I like, the author poked fun at the supposed notion that mathematicians are intransigent and stubborn, failing to accept new ideas in a timely fashion (this is not merely an outside opinion, there are some insiders who feel the same way… quite strongly in fact. See Doron Zeilberger’s opinion page for instance). However, as someone who is about to get a PHD in mathematics and an amateur mathematics historian, I would like to voice my polite disagreement with Mr. Weinersmith’s premise.

This is the message I posted on the comic’s Facebook page:

“As someone about to get a PHD in mathematics, I can attest that the basic premise of this comic is wrong. Mathematicians have always been much quicker to accept new advances and shifts in paradigm faster than their contemporaries in other fields. The only times when acceptance of new results, even paradigm shifting ones, were slow to be accepted by the mathematical community are those where the result was poorly written or poorly presented (for example, Cantor’s work on cardinality, Brun’s sieve theory, Ramanujan’s work before Hardy, Heegner’s solution of Gauss’s class number one problem, and most recently and still unresolved: Shinichi Mochizuki’s purported proof of the abc conjecture)

Edit: to give a positive example, consider the proofs of Fermat’s Last Theorem and more recently, the Poincare conjecture. These two are considered two of the most difficult mathematical problems in history, and when their solutions were presented, it took only a few years for the mathematical community to verify and accept their correctness. Even more recently, Yitang Zhang’s manuscript containing the proof of the existence of infinitely many primes which are within a bounded distance from each other was accepted in JUST THREE WEEKS by one of mathematics’ top journals, even though Zhang was at the time completely unknown and in particular was not known to have done any work in number theory.”

I would like to elaborate even further on my comments. Not only are mathematicians not intransigent as suggested in the comic, mathematicians are likely to be the group in academia which is the most willing to share their ideas and accept other people’s ideas (this is a broad stroke, there are certainly many people who arbitrarily dismiss people’s work, as anyone who has faced a grouchy referee when submitting a paper can attest) . The lightning fast acceptance of the two big advances on the bounded gap problem should serve as a testament to this. Both of the main players, Yitang Zhang and James Maynard, were at the time more or less completely unknown. Their ‘lowly’ status did not prevent their work from being recognized, almost instantly in fact, by some of the biggest experts in the field (including Andrew Granville and Terence Tao). This seems unlikely in many other areas, especially as one gets further away from pure science.

This is not to say that mathematicians are just more progressive and forward-thinking in general. Social attitudes among mathematicians, while probably better than the general population, is certainly not stellar, as a recent paper by Greg Martin points out.