Cloak & Dagger Thread

[Cyneheard]

This post is going to be a placeholder for up-to-date C&D information/links.

We have three options for a C&D file:
1) Dropbox shared Excel 2003 file. I will create a shared folder that would allow ANY team member to edit the file, while letting me use the existing C&D structure and build a lot of automation. I prefer this because it's a) built, b) easy to create straightforward summaries, and b) it's more powerful software for analysis.

NOTE: This shared folder will integrate with existing Dropbox accounts. I.e., if you have Dropbox, and use it as a folder in Windows Explorer, the Team Gillette folder will show up under your Dropbox files on your home machine.

Protocol for gaining read-write access to the C&D file would be: PM or e-mail my RB address at gmail, which is "cyneheardegypt" for access with the e-mail address you'd want put in dropbox. Everyone would have access to the public file link below:

Current C&D Data:
RB Demogame 1 C&D Spreadsheet

2) Google Docs. This is somewhat easier for collective data entry, but as a data analyst I'm always concerned about presentability and usefulness of the final data, and I don't trust it to preserve data very well.

3) Another suggestion.

[Cyneheard]

Note: I put a deadline on the poll of a week, as once the game's started, changing the master C&D file is asking for trouble.

What is "Cloak & Dagger" Analysis, and why do we care?

Reader's Digest Version: Gaining as much information possible about our opponents through in-game data sources:

[Data Collection] ---> [Understanding of Civ4 Mechanics] ---> [Inferences about Civs] ---> [Actionable Information]

After all, the whole point of data analysis is to make the best decisions possible. And Civ4 provides an absolute wealth of information that we can digest, although some things require more analysis than others, and some pieces of information are less certain than others.

Uncertain info example:
If one civ teched The Wheel, and another civ techs Agriculture on the same turn, it may not be possible to tell which civ teched which tech.

Key data sources (not an exhaustive list):
Civstats (score changes, and particularly who is logged in when the score changes, are massively useful)
The F9 screens:
- Graphs earned through espionage
- The Demographics Screen (I'll go through the Demogs screen point-by-point later)
- Top 5 Cities (Data given: Pop of top 5 cities, date of founding, and wonders)
The F8 Victory screen:
- Mostly population data
The Espionage screen (requires City Visibility to be fully useful):
- Especially Sabotage Production to track city builds, although there's a lot of info available there.
Visible information:
- Resources (requires tile visibility; matters most for strategic resources)
- City buildings (requires city visibility)
- Known units

[Cyneheard]

Understanding of Civ4 Mechanics

SCORE.
Your score is the sum of 4 factors:
Population.
Land.
Wonders.
Tech.

Population: Each pop is worth a fixed amount of points, based on the map. We can find this out on the first turn of the game.

Land: If we owned every land tile on a map, then we would have 2000 points. Each land tile is worth (2000 / # Of land tiles on the map). We can find this out on the first turn of the game. Mouse over our civ's score, and this should pop up:



In this game (and this shot is from T0):
Each Population point is worth 5000 / 2107 = 2.37 points
Each Land tile is worth 2000 / 2054 = 0.97 points.
The 5000 and 2000 are defined in the XML; 2107 and 2054 are map-dependent. The more watery the map is, the higher the pop score is relative to the land score. T-Hawk got to about 3x the map's "Max" population in his Marathon Sushi high-score game. But corporations break game mechanics in a lot of ways.

Land tiles only count when they've been owned for 20t, so I usually mark on a chart when I expect a civ to earn land points from settling their capital, expanding borders, etc. etc. If you look at the "Score" page here, and scroll up, you'll notice that on T20 and T21 Rego put in "9 (22) land", meaning: 9 land tiles earned 22 score on that turn for that civ.

Wonders: Each wonder (National OR World, so the Palace counts) is worth (1000 / 62) = 16.13 points per wonder. So wonders 1 - 7 are each worth 16 points, wonder #8 is worth 17, etc. etc.

Tech: Each tech is worth 6*that tech's era value (1 = Ancient, 2 = Classical, 3 = Medieval, etc. etc.) (EXCEPTION: the first tech is worth 5 points, not 6, because it's technically 5.99 points * the tech's era value). So we start with 11 points from tech, and earn 6 per Ancient Tech, 12 per Classical, and 18 per Medieval tech.

Other mechanics information:

Civfanatics is your friend. They have by far the most comprehensive information about game mechanics of any civ site. NOTE: Not all of these are C&D-related, but they're all useful at different times.

The big 2, which will each get their own posts:
Inner Workings of the Demographics Screen

Espionage Mission Costs

Various useful references:

City Upkeep Explained

Great People Bulbing Preferences

Tech Research (Key points: the 20% "OR" bonus, how the Known Civ Bonus actually works)

I wish I had a good explanation of the combat mechanics; the CFC combat mechanics page is wrong in several aspects, especially with how it deals with first strikes and damaged units (not entirely their fault; the mechanics changed on damaged units since that article was written). But combat mechanics aren't the point of C&D, so I'll leave them for another day.

[Cyneheard]

The Demographics Screen



This is an early demographics screen from PBEM22. NOTE: Espionage was removed from this game, so we don't get +4 espionage in GNP.

With 4 opponents, we have 3 pieces of information:
Rival Best
Rival Average (NOTE: Civ basically always rounds down. This is important for, say, early GNP parsing)
Rival Worst

Rival Average * 4 - (Rival Best + Rival Worst) = Average of 2nd and 3rd place in a given Demographics field, after taking rounding into consideration. So we will always know:
#1
#4
Sum (#2 + #3) for every field.

How do we track this information?
- Take the demographics screenshot every turn. On Turn 0, ideally, we want to take the screenshot after each different team settles.
- Derive the changes that happened to each civ over the previous turn. Example:

Using our example screen above.
Let's say on T6, Nakor's stats were:
Score = 28
GNP = 11
Power = 2k
Now, on T7, it's:
Score = 34
GNP = 13
Power = 4k

We were all Joao of the Inca, so we started Myst/Agri
Since his score increased by +6 points this early, that can only be a tech.
Since his GNP is 2 higher, his current tech selection probably has a pre-requisite.
His power increased by 2k, which should be from the tech.
As Mining could be completed in 6-7t, increases power by 2k, and usually leads to teching Bronze Working, which is a tech with a pre-requisite, I surmise that Nakor teched Mining.

Note that we used about 4 different sources of information to figure out Nakor's tech. This is why tech deduction, even when opponents aren't trying to deceive us, can be dicey.

The various fields, and what we use them for:

GNP:
Total commerce + total culture + total espionage + (total research * research bonus) ***8211; expenses.

The "research bonus" is the only complicated thing here, as that includes:
Known Civ Bonus (30% * Number of Known Civs with the Tech / Number Of Civs In Game)
The 20% pre-req multipliers.

Early on GNP only varies for a few reason:
- Who's working +1 commerce?
- Who's working a tech with a pre-req? This is generally worth +2 GNP per pre-req.
- We have 2 Creative opponents (Sury, Willem) and 2 non-Cre opponents (Huayna, Mansa), so parsing GNP is going to be a little dicey. This is also worth +2 GNP for the Cre civs while they only have one city.

Mid-game and late-game, we use graphs instead to figure out what civs are doing, as rough qualitative info is just as good.

MFG:
Hammer production. Includes all hammer modifiers that aren't situational (i.e., forges but not OR)

Crop Yield:
Food production. Straightforward.

Soldiers:
Detailed information on soldiers is here: Inner Workings of the Demographics Screen

Short version:
Every 2 pop points = 1k power
Units are usually 1k power per unit of strength (+50% vs. Melee is valued at +1k power)
Buildings and techs vary

Therefore, the soldier count is a useful check on:
- Population count for our opponents
- Unit builds
- Techs, especially earlier. We should know for sure when our opponents tech Bronze Working (8k) and Archery (6k).

However, because Soldiers combines three different things, we have to be careful that we aren't confusing warrior builds with, say, tech power.

Land area is 1000km / tile. This will help us determine who's got water in their initial BFC (land points, which I'll discuss when I talk about score separately, is more important, but less immediate, as it takes 20t of holding land for it to count).

Population deserves its own post.

Approval Rate:
(Happy in all cities) / (Health + Unhealth in all cities)

Mostly marks how well civs are using their happy cap. A high approval rating means that your civ should be growing or whipping more. One below 50% means you've got problems.

Life Expectancy:
(Health in all cities) / (Health + Unhealth in all cities)

Value T0/1:
- Flood Plains, (Jungle, which is rare), Forests, and Expansive all go into early health. So we should know how many forests people have in their BFC (one reason why grabbing Demogs after EACH civ plays is key)
Value after:
Not much.

Exports - Imports:
Tells us who's trading internationally. I don't pay much attention to this one.

[Cyneheard]

Espionage Costs

This isn't about an espionage economy, this is about how Espionage data is useful. I've mostly cribbed other people's notes on this one, so it's less useful than I'd like. Especially since the CFC guide doesn't include the Sabotage Production mission!

If we have visibility on a city, then we can learn the espionage cost of a mission in that city.

Sabotage Production:
Every turn, we can calculate how many hammers a city has on its production item. If we track this over time, then it's fairly easy to get things like:
- Hammer production in a city
- Guess on what they're building (workers/settlers, Ind wonders tend to act differently from generic buildings/units)

Formula: (Sabotage Production * (Their EP + 100)) / (Investigate City * 3).
"Their EP" is visible by, among other mechanisms, mousing over our opponent's score. It'll give us two numbers like so: "X/Y," where X is how many EP we spent on them, and Y is how many EP they've spent on us. Sabotage Production and Investigate City are through the Espionage window (Ctrl + E)

An interesting one is this:
Steal Treasury:
The amount of gold you can steal is based on a civ's Gold supply * GenericEspionageFactors * City Pop / Civ's Total Pop. So it's possibly worth digging around the code (where's T-Hawk when you need him?) to figure out how Steal Treasury can be calculated to get either:
- A civ's total gold supply
- A civ's total population
(Note: we can't get both from this data, as we only have one equation to work with). Since we'll know the pop of cities we have visibility on, this is potentially very useful.

[Cyneheard]

Collecting Data

I think this is going to be a discussion piece, not a treatise.

However, information that we'll always need:

- The Demographics screen should be collected every single turn. And we should make it as obvious as possible what turn the demographics shot was taken. I'm not sure how we want to organize them, though.
- Graphs when we first get them. Remember, graphs are always at least 1t out of date. Sullla was very good at analyzing graphs; I've not done as much with them.
- Espionage data, especially when we have City Visibility.
- Civstats score changes. Useful pieces of data:
- Score change @ Turn Roll (Y/N?)
- Amount of score increase or decrease. Who was logged in when it happened, if mid-turn? This should make it easy to tell when a city is captured/built/whipped, and if captured, whodunnit.

Basically, every screenshot that we take should be checked over for data, in an ideal world, but those are things that we most want. I think.

[Cyneheard]

Population Count:

The Information We Want:
a) How many cities does Civ A have?
b) What is Civ A's total population?
c) What is the population in each of Civ A's cities?

The # of cities an opponent has can be found two different ways:
- Watching civstats for midturn score increases. As huts and tech trading are off, I'm 99% sure that new cities are the only mid-turn score increase possible.
- Once we've met a civ, we can pull up the diplo window and count their # of cities that way.

Determining Civ A's total population:
- Score includes a population component. More details are in the Understanding Civ4 Mechanics post above.
- Power rating includes (Population Points / 2).

The population in each of Civ A's cities:
- The Top 5 Cities screen will give us the population of the 5 "greatest" cities in the entire game. With 5 civs, I expect this to essentially be our capitals, at least until the wonderspamming starts happening. There are rules behind the "greatest" cities, but I don't really know them. I know culture and population matter. So large wonderspammy cities show up, as we'd expect.
- The demographics screen gives us a population value for each civ. It is the sum of each city's population value, which follows this pattern:

Size --> population value
1 --> 1k
2 --> 6k
3 --> 21k
4 --> 45k
5 --> 90k
6 --> 150k
7 --> 232k
etc. etc.

Example:
Known information about Civ A:
5 cities.
19 population.
Their demographics population value is 286k. (If I use "k" after a number, this is the number I'm talking about. If I don't use "k" after a number, I'm talking about either a city size or total civ-wide population)

By "capital", I mean "largest city", although that's usually the capital, of course.

Let's try a capital of size 7 = 232k.
That leaves us with 54k across 4 cities and 12 pop.
If we try 4/x/x/x, we can't get to 54k pop: 3 cities can never generate the final 9k in pop.
If we try 3/3/2/2, we get the correct 4 cities and 54k in pop value. BUT we only get 10 pop, when we need 12 based on our other data. So a capital of size 7 doesn't work.

Therefore, let's try a size 6 capital = 150k.
That leaves use with 136k across 4 cities and 13 pop.

City #2: Size 5 = 90k. That leaves us 46k across 3 cities and 8 pop.
4/1/1 = 47k, 6 pop. No.
3/3/2 = 48k. No.
3/3/1 = 43k. No.

So City #2: Size 4 = 45k. Leaves us with 91k across 3 cities and 9 pop.
4/4/1 = 91k, 9 pop.

Conclusion:
Capital = Size 6
City #2 = Size 4
City #3 = Size 4
City #4 = Size 4
City #5 = Size 1

Potential Issues as Time Passes:
1) As we saw above, # of cities + demographics pop value isn't always enough data to figure out a civ's city sizes.

The most common version of this issue is the fact that both:
2/2/2/2 and 3/1/1/1 are 24k pop value.

Even if we know the total # of pop points a civ has, these near-equivalencies are still problematic: this is a heavily manual process. And if, for instance, we had an opponent with 310k power, 9 cities, and 25 pop, we couldn't automatically tell which of these two distributions was true:
7/3/3/2/2/2/2/2/2 (i.e., the 286k/17 pop + 24k/8 pop situations. 1*232k + 2 *21k + 6*6k = 310k. 7 + 2*3 + 6*2 = 25 pop)
6/4/4/4/1/3/1/1/1 (i.e., the other options: 1*150k + 3*45k + 1*21k + 4*1k = 310k. 6 + 3*4 + 3 + 4*1 = 25 pop)

2) More cities = more possibilities. You saw how much it took to derive 5 cities. When there are 12, 15, 18 cities a civ, it could get worse.

3) Whips mean that cities can increase by 1 pop on a given turn, or decrease by 1 to N pop, so knowing last turn's pop in each of Civ A's cities only tells us so much about what this turn's pop is in each of Civ A's cities.

BUT:
However, eventually we should get visibility on opposing cities, which means instead of doing all of this complicated legwork we simply...look at the map.

[Cyneheard]

Translating Inferences about Civs to Actionable Information:

That's everyone else's problem. :neenernee

Once you've got the data, you've got to use it. Diplomacy, planning, etc. etc.

And remember the first rule of data analysis:
Bad data is worse than no data.

Of course, that doesn't mean imperfect data is worse than no data, but my MO is that I'd rather say "I don't know" or be clear about our uncertainty in the C&D data than making bad assumptions that lead us down the wrong path.

Not sure there's much else to say here, other than this is WHY C&D matters.

[Cyneheard]

A Post-Script.

How To Screw With Other People's C&D

This is really only possible with one thing:
Tech. The main C&D "outputs" (Score is an input, not an output):
- Population
- Techs
- Power
- Espionage/Foreign Builds

Techs are by far the most uncertain, as techs can only be identified by:
Era, through score changes
Power changes aren't unique to individual techs OR to teching
Beakers put into a tech aren't directly visible...and it's possible to deceive opponents on this one.

There was an interesting example of misdirection in PB2, from the original Rome player. I'd have to find the link, although IIRC Rome had to also change their city micro to do it too, which is IMO way too expensive for keeping people from figuring out what you teched in the early game where every hammer is insanely precious.

What we may want to consider in a micro plan: can we delay a cheaper tech to look like a more expensive one without any cost to our civ? If so, then we probably should, if only to confuse our opposition. And we should expect people to do the same against us.
If we're teching Mining first, it's 50 base beakers. If we have it take as long as Agriculture would, and another civ finishes Agri that turn, then our opponents, even with good C&D will realize:

One civ finished Mining on a delay to screw with them, and the other finished Agriculture. But they may not know which of the civs went Agri and which one is screwing with them. It has some value, but it's hard to REALLY do right.

[mackoti]

Good Job Cyneheard.Perhaps i will learn a litle bit myself abot that.