Stanford GSB Sloan Study Notes, Week 7 (37), Spring quarter
This week was completely overrun with two homework projects that refused to surrender even when groups of Sloans & MBAs spent hours sitting in the room and cranking through logistic regressions and ROI analysis for marketing channels of a pharma company, or trying to calculate the optimal inventory cost of the supply chain of HP injet printer factory. Also, a real dataset of 90,000 users of mobile phone users came in, which will be a basis for one of our final projects on understanding churn – once we chew ourselves through it.
And in the Town Hall with Madhav Rajan, Associate Dean of Academics we learned that the incoming MSx 2014 class will have quite a few improvements to their class schedule, with even more room for electives in Winter & Spring – congrats! Make sure you’ll fill those slots with quant analysis under the California sun this time next year.
Covered in this issue:
- Milgram’s Small world problem and its modern developments
- Recency/Frequency/Monetary Value & Churn analysis
- Measuring sales force performance and forecasting tricks (and a video sampler for a litmus test for if you should be in sales)
- Organizational blueprints for startups
- Guests from: Google, VMWare, Progreso, Ariat
OB322: Networks (Hasan)
Article: The Small-World Problem (Milgram’s original text from Psychology Today in 1967)
- the Small-world experiment article on wikipedia gives a nice quick summary (and animated maps!)
- If you had completely random connections between people, you would get to anyone else on the planet in much shorter path than “six degrees of separation”. In a continuum of “high clustering” (long paths) -> “completely random” (shortest paths), just 1% of connections have to be random to achieve a small world.
- Same pattern: WWW, e.coli, US power grid…
- A lot of modern research on small world problems including Random Rewiring above: Duncan Watts (Microsoft Research). 20k+ citations on his papers.
- When choosing someone to pass off a message to someone outside your network, best strategies look for (based on CS research):
- demographic proximity
- high degrees (in/out)
- high betweenness
- In real human networks, demographic beats others (in HP Labs, passing a message by hierarchy: 4 hops, by office location: 6 hops, via the most connected: 16 hops – started bouncing between biggest brokers)
- Preferential attachment process – in which some quantity, typically some form of wealth or credit, is distributed among a number of individuals or objects according to how much they already have, so that those who are already wealthy receive more than those who are not.
- In case of social networks: if there are three interconnected nodes, an incoming node connects to this cluster thorough one of them by random – say to node C. But after this connection, it is just a little bit more likely that another incoming connection is done to C (as opposed to A or B), because .. it looks more connected.
Guest: Mark Rivera (People Analytics, Google)
Topic: Science of HR at Google
- notes remain private by request of the speaker
- Minimal group paradigm research (for Day 1 employee experience)
MKTG 365: Marketing Analytics (Narayanan)
- RFM analysis is the oldest database marketing technique, looks at Recency, Frequency and Monetary Value of individual past customers to make targeting decisions – and assigns new purchase / response probability to each bucket.
- Makes use of powerful metrics of customer behaviour that are contained in most house files (easy to access). Also easy to understand and communicate – does not require training in statistics (Excel and averaging enough)
- Recency (days since purchase) curve against new purchase probability can have different shapes (decreasing, hyperbolic, cyclical), but relationship is clear. The curve shape usually doesn’t matter, because the groups will be stack ranked based on their probability (not actual recency)
- In any case RFM analysis requires a lot of data. In case of independent analysis, without enough data, the size of groups will be very different. With sequential ordering (first R, then F inside that, etc) the groups are always the same size, but without enough data a few outliers can notably mislead your results – and RFM 111 is not necessarily responding better than 211 (intuition breaks).
- RFM analysis difficult to implement
- for finer segmentation than quintiles – even deciles problematic (30,000 observations: in quintiles, 125 cells, 250 observations per cell. Same data in deciles: 1000 cells, only 30 obs. per cell!)
- to add additional explanatory variables (demographics, etc)
- As a response to limitations, use logistic regressions, decision trees or neural networks – but RFM still provides the intuition behind the more complicated statistical models
Topic: Churn Analysis
- Sun spot activity and stock market – a well known case of overfitting (just 100 years of data for both)
- Characteristics of logistic regressions
- Pseudo R^2 is even in best cases significantly lower than the R^2 of linear regressions
- the coefficients for independent variables don’t have a direct interpretation as with linear regressions (elasticity, %% impact, etc).
- Coefficients show the change of abstract “utility units” (utils) – so the absolute value doesn’t say much, but the change in it (“how does the probability of churn change when revenue changes by 1%”) – you can calculate the elasticity.
. logit churndep revenue-retcall
Iteration 0: log likelihood = -26991.574
Iteration 1: log likelihood = -26166.913
Iteration 2: log likelihood = -26150.176
Iteration 3: log likelihood = -26150.169
Iteration 4: log likelihood = -26150.169
Logistic regression Number of obs = 38941
LR chi2(66) = 1682.81
Prob > chi2 = 0.0000
Log likelihood = -26150.169 Pseudo R2 = 0.0312
------------------------------------------------------------------------------
churndep | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
revenue | .0019635 .0007981 2.46 0.014 .0003992 .0035278
mou | -.0002809 .0000496 -5.66 0.000 -.0003782 -.0001836
recchrge | -.0031225 .0008888 -3.51 0.000 -.0048645 -.0013806
…………….*snip* ……..
setprcm | -.0963204 .0405136 -2.38 0.017 -.1757257 -.0169152
setprc | .0006204 .0002827 2.19 0.028 .0000662 .0011745
retcall | .7936977 .1945818 4.08 0.000 .4123244 1.175071
_cons | .1497005 .0952738 1.57 0.116 -.0370327 .3364336
------------------------------------------------------------------------------
Note: 0 failures and 1 success completely determined.
. margins, dydx(revenue)
Average marginal effects Number of obs = 38941
Model VCE : OIM
Expression : Pr(churndep), predict()
dy/dx w.r.t. : revenue
------------------------------------------------------------------------------
| Delta-method
| dy/dx Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
revenue | .0004702 .0001911 2.46 0.014 .0000957 .0008447
------------------------------------------------------------------------------
. margins, eyex(revenue)
Average marginal effects Number of obs = 38941
Model VCE : OIM
Expression : Pr(churndep), predict()
ey/ex w.r.t. : revenue
------------------------------------------------------------------------------
| Delta-method
| ey/ex Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
revenue | .0582378 .0236117 2.47 0.014 .0119597 .1045159
------------------------------------------------------------------------------
- Interpretation
- marginal effect (margins, dydx): for people whose revenue goes up by $100, their probability(churn) goes up by 100*.0004702=4.7% (4.7 of additional people of every next 100 churn)
- elasticity (magins, eyex): for 1% change in revenue, the churn increases compared to current level by 0.05%
- Stepwise regression: runs a regression on all variables, orders by p-value, drops the highest (least significant) and reiterates until only significant variables remain.
- Stata: stepwise logit churndep revenue-retcall, pr(0.05)
- Finding exact churn predictions is hard, more reasonable relative questions to ask: “which users are more than 75% more likely to churn than average”. Generate a lift variable = predicted_churn / average_churn. Then filter by lift>1.75
OIT 262: Operations (Plambeck)
Topic: Suppy Chain Inventory
- Point estimates on expected demand are of little use, you need estimation on their accuracy too (is the estimate of 2000 distributed with std deviation of 100 or 700?)
- Demand forecasting and calculating demand forecast errors
Topic: HP DeskJet
- Delay the customisation/localizations of products in the cycle as long as you can
STRAMGT 351: Building and Managing Professional Sales Orgs (Lattin/Levine)
Reading: What can we really manage? (Ch 2)
Book: Cracking the Sales Management Code: The Secrets to Measuring and Managing Sales Performance
- In the argument around if sales is more of art or science, superstar sellers seem to prove the tilt towards creative arts. In case of sale management, the high performers are for sure more scientific (analysis, not gut; rigorous goal setting, tracking; formal meetings, formal agendas). Consider if your top sales guys are really the best material for promotion to managerial roles?
- Risk of the “make your numbers” mantra: people can still only manage people and resources, not concepts or numbers. Can you really manage your kids’ grades, or can you manage their behaviour and create an environment where they are more likely to improve the grades?
- Beware of the death by reporting:
- $ Profit, $ Revenue, % Share of Market, Customer Satisfaction, Unit Volume, % Sales calls advancing, % Customers contacted, Day Ramp-up Time, % New Product Sales, % Customer Retention, % Share of Wallet, # New Customers, % CRM usage, % Account Plan Completion, # Prospecting Calls, # Accounts per Rep, $ Spent on Training, # Reps per Manager…?
- No real industry standard or best practices, even inside a single niche
- A survey surfaced 306 different metrics used by sales organisations. A sanity check of each one with sales managers across companies:
- 17% of those deemed highly manageable, all related to sales activities and activity metrics.
- 80%+ of metrics used can only be indirectly influenced by managers.
- The latter, indirect category is in practice split 1/3 business results (revenue, CSAT) and 2/3 sales objectives (share of wallet, customer retention…).
- Reverse-engineered chain of Sales Metrics: To achieve “3% increase of revenues” (business result) -> drive towards “4 new customers per quarter” (sales objective) -> by managing “16 more prospecting calls per quarter” (sales activity).
Blog post: Building a Culture That Works by Peter Levine
- In trusting people, engage them in your thought process:
- Don’t just share a decision, but the pros/cons discussed and path why we came out on the side we did
- put yourself in a teaching position for your org (behave like your favourite professor)
Topic: Forecasting
Guest: Bryan Cox (VP of Global Support Renewals, VMWare)
- Common issues with forecasting
- Asymmetric incentives (quota always more important). Forecasting should be at least “marker for evaluation”
- Asking for “blood commit” numbers first anchors the forecasts lower
- Sandbagging is safer than shooting over
- Anchoring to quota: it is some arbitrary number given from above
- No explicit and regular feedback to sales reps on their forecasting performance
- Easier to be a good forecaster when you’re crappy at sales (more time, lower numbers, less absolute variance)
- Stops at “a $ number”, no actionable or helpful conclusions on what the managers and sales people should do knowing the number, where to allocate the resources
- Data is sliced by reps and managers, not pooled together for comparative views (no sales operations, analytical people crunching numbers)
- Running scientific experiments on employees to get reliable performance data can be unfair (and borderline creepy)
- Hermann Brain Dominance Instrument – for sales team evaluation
- Great sales people: D. Imaginative thinking: Visual, holistic, intuitive, innovative, and conceptual.
- Great forecasters: A. Analytical thinking: Auditive,logical, factual, critical, technical and quantitative.
- To driver commitment and accountability: do it in person & among peer group
- Try also to track “upside”: deals not forecastable yet, but potential to move up
- More risk in paying money for forecasting performance (tension with actual sales achievement), but a great basis for promotion. Sales performance given for all, but best forecasters understand the business, can teach others they methods…
- The more linearity you can impose on a hockey stick process (like sales in quarter) – the less surprises you get
- “Sale postponed is sale lost” – reason not to strongly incentivise forecasts (and exceeding them)
Movie: Glengarry Glenn Ross
- Litmus test if you should be in sales or not. Great sales people love this movie. If you get uncomfortable: wrong job.
Case: Progreso Financial
Guest: Sam Ulloa (VP Sales, Progreso)
- Legal requirement: if you have employees physically in some location every day and you tell them ahead what they need to do – you need to pay them a base salary (not just incentives on sales)
- Sanity check your incentive model: after setting all steppings and levels the intended 100% payout (base + incentives) require the unit sales you need? Can get out of sync: full payout at 40 contracts, if your business plan needs 50 per person…
- When building an early, “hunter” salesforce, watch out for when to augment it with customer support. Do not waste hunter’s energy and attention and get better service for existing clients from people specifically incentivised to provide that (not sell)
- When not ready to start a company, don’t go to banking or consulting to learn more – go to sales. As and entrepreneur you will always be selling your vision and product – to first employees, investors, clients…
STRAMGT353: Formation of New Ventures (Holloway/Morgridge/Chess)
Article: Organizational Blueprints for Success in High-Tech Startups by Baron & Hannan (Stanford 2002)
- From Stanford Project on Emerging Companies (SPEC) – also referenced from Noam Wasserman’s Founders Dilemmas (see the reading notes post here)
- Dimensions of employment blueprints across 200 startups viewed (thorough the dot com boom):
- Basis of attachment & retention: compensation (“money”), qualities of work (“work”), work group as community (“love”)
- Criterion for selection: skills, exceptional talent/potential, fit with team/organization
- Means of Control & Coordination: direct monitoring, peer and/or cultural, reliance on professional standards, formal processes & procedures
- Across 3 dimensions, there are 36 possible combinations – in reality 5 patterns emerge. Founder quotes:
- Star: “we recruit only top talent, pay top wages & give them resources and autonomy they need to do their job” (Work/Potential/Professional from above dimensions)
- Commitment: “I wanted to build the kind of company where people would only leave when they retire” (Love/Fit/Cultural)
- Bureaucracy: “We make sure things are documented, have job descriptions and pretty rigorous project management techniques” (Work/Skills/Formal)
- Engineering: “We have a skunk-works mentality and the binging energy is very high” (Work/Skills/Cultural)
- Autocracy: “You work, you get paid” (Money/Skills/Direct)
- Replacing founder CEO changed blueprints in 3/4 cases, in 40% of cases – by two or more dimensions.
- Engineering & Bureaucracy models are the closest (only differ on means of control), and also there is the most notable shift towards latter when non-founder CEOs take over.
- Blueprints strongly correlated with actual admin overhead in organization, in 50, 500, 1500 people size alike: in Commtiment based models, share of managers and administrators of total staff was 11-17%, in Autocracy & Bureaucracy models, up to 37%!
- Commitment based models most likely to go public and Star-based least likely. Yet, Star-based organisations grow post-IPO market cap 80% faster than next best.
- Compared to Engineering model based company, Autocratic is 130% more likely to fail and all other less likely to fail (Commitmend-based -100%)
- Beware of the strong focus on “fitting in” in Commitment model. Notably limits diversity (gender, viewpoints..) becoming natural for the culture.
Case: Cisco
“Guest”: Prof Morgridge (as ex-CEO of Cisco)
- Why start a company at a university?
- Basic research
- Young minds
- Free resources/facilities
- Latest Tools & Equipment
- Answers are one department away
- Risk/Failure tolerant
- Low cost (no mortgages yet)
- Long runway
- Beta test audience who cares
- What should a new CEO focus on in first 90 days when talking to people? Not just isolated view (their skills, their views, their obstacles), but a network view (who do they look up to, who do they think are top performers/potential)
- Non-founding CEO should remember that investors invested in the founder, not him/her
- “You can always change the culture. You just need three bad quarters.” (Cisco was built on Dr Pepper. Started charging for soft drinks after 2008)
BOOK: Crossing the Chasm: Marketing and Selling Disruptive Products to Mainstream Customers
Case: Ariat
Guest: Beth Cross (Founder & CEO)
- #1 horse riding boot brand in a world (after entering the “old boys club” from zero in early 90s)
- In consumer space, great to work outside designers to get fresh perspectives
- With a new & differently positioned product, you have to differentiate with the margin you’re leaving to the retailer to get in the door (50%, when others do 20%). Presumes premium pricing & positioning.
- First year sold 1000 pairs and “seeded” (gave away to clerks for their own use) at least 500. The clerk stands between you and the sale, he/she needs to be a believer.
- Visualize your company as a path over time, not just today or not the end point. One way: map out products, channels, marketing, geographies, capabilities, customers, technology as concentric circles and show in which order you would like to expand coming from inside.
—-
For more posts on the Stanford GSB Sloan life – see the table of contents here.