Showing posts with label graph hbar. Show all posts
Showing posts with label graph hbar. Show all posts

Jul 29, 2017

OLS regression: overfitting and dichotomization

Babyak (2004) demonstrates a couple of aspects of OLS regression, making use of the following simulations:
// Figure 1
set seed 1

// Create file to store simulation results
tempname foo
postfile `foo' b using clt, replace

// Simulate analyses
forvalues i = 1/10000 {
    drop _all
 qui set obs 100
    generate x = rnormal()
 generate e = rnormal()
    generate y = .4 * x + e
    qui regress y x
 local b = _b[x]
 
 post `foo' (`b') 
}
postclose `foo'

// Open results from simulations and plot
use clt, clear
histogram b, freq ytitle("Frequency of b value") xtitle("Values of b") name(figure1, replace)

// Figure 2 set seed 1 // Create file to store simulation results tempname foo postfile `foo' r2_50 r2_100 r2_150 r2_200 using overfitting, replace // Simulate analyses forvalues i = 1/10000 { drop _all set obs 10000 generate y = rnormal() foreach i of numlist 1/15 { generate x`i' = rnormal() } foreach j of numlist 50 100 150 200 { preserve sample `j', count qui reg y x* local r2_`j' = e(r2) restore } post `foo' (`r2_50') (`r2_100') (`r2_150') (`r2_200') } postclose `foo' // Open results from simulations use overfitting, clear // Plot twoway (kdensity r2_200) /// (kdensity r2_150) /// (kdensity r2_100) /// (kdensity r2_50) /// , ytitle("Percent of samples") /// xtitle("R-square value from regression model") /// xlabel(0 (.1) .6) /// ylabel(0 (2) 20) /// legend(order(1 "ca. 13 cases/predictor ({it:N} = 200)" /// 2 "10 cases/predictor ({it:N} = 150)" /// 3 "ca. 7 cases/predictor ({it:N} = 100)" /// 4 "ca. 3 cases/predictor ({it:N} = 50)") /// pos(2) ring(0)) name(figure2, replace) // Figure 2 set seed 1 // Create file to store simulation results tempname foo postfile `foo' r2_50 r2_100 r2_150 r2_200 using overfitting, replace // Simulate analyses forvalues i = 1/10000 { drop _all set obs 10000 generate y = rnormal() foreach i of numlist 1/15 { generate x`i' = rnormal() } foreach j of numlist 50 100 150 200 { preserve sample `j', count qui reg y x* local r2_`j' = e(r2) restore } post `foo' (`r2_50') (`r2_100') (`r2_150') (`r2_200') } postclose `foo' // Open results from simulations use overfitting, clear // Plot twoway (kdensity r2_200) /// (kdensity r2_150) /// (kdensity r2_100) /// (kdensity r2_50) /// , ytitle("Percent of samples") /// xtitle("R-square value from regression model") /// xlabel(0 (.1) .6) /// ylabel(0 (2) 20) /// legend(order(1 "ca. 13 cases/predictor ({it:N} = 200)" /// 2 "10 cases/predictor ({it:N} = 150)" /// 3 "ca. 7 cases/predictor ({it:N} = 100)" /// 4 "ca. 3 cases/predictor ({it:N} = 50)") /// pos(2) ring(0)) name(figure2, replace)
// Figure 4 (actually Table 1) clear set seed 1 // Create file to store simulation results tempname foo postfile `foo' n correlation typei using dichotomization, replace // Simulate analyses forvalues i = 1/10000 { drop _all foreach j of numlist 50 100 200 { foreach k of numlist 0 .3 .5 .7 { qui drawnorm x1 x2, /// n(`j') /// corr(1, `k', 1) cstorage(lower) /// clear generate e = rnormal() generate y = .5*x1 + 0*x2 + e qui sum x1, detail generate x1s = (x1 > r(p50)) qui sum x2, detail generate x2s = (x2 > r(p50)) qui reg y x1s x2s local typei = _b[x2s]/_se[x2s] local sig = (abs(`typei') > 1.96) *di _b[x2s] _skip(5) _se[x2s] _skip(5) `typei' _skip(5) `sig' post `foo' (`j') (`k') (`sig') } } } postclose `foo' // Open results from simulations use dichotomization, clear collapse typei, by(n correlation) graph hbar typei, over(n, relabel(1 "{it:N} = 50" 2 "{it:N} = 100" 3 "{it:N} = 200")) /// over(correlation, relabel(1 "{it:Corr(x{sub:1}, x{sub:2})} = 0" /// 2 "{it:Corr(x{sub:1}, x{sub:2})} = .3" /// 3 "{it:Corr(x{sub:1}, x{sub:2})} = .5" /// 4 "{it:Corr(x{sub:1}, x{sub:2})} = .7")) /// ytitle("Type I error rate") yscale(alt) ylabel(, format(%6.2f)) /// name(figure4, replace)

Reference

Babyak, Michael A. 2004. "What You See May Not Be What You Get. A Brief, Nontechnical Introduction to Overfitting in Regression-Type Models." Psychosomatic Medicine 66(3):411-421. doi: 10.1097/01.psy.0000127692.23278.a9

Aug 20, 2015

Random graphs (51): Bar graph


// Data from 
// de Mouzon, Jacques, Veerle Goossens, Siladitya Bhattacharya, José Antonio 
//     Castilla, Anna Pia Ferraretti, V. Korsak, Markus Kupka, Karl-Gösta 
//     Nygren, Anders Nyboe Andersen, and the European IVF Monitoring
//     Consortium for the European Society of Human Reproduction and Embryology.
//     2012. "Assisted Reproductive Technology in Europe, 2007. Results
//     Generated from European Registers by ESHRE." Human Reproduction 27(4):
//     954-966. doi: 10.1093/humrep/des023
// Ferraretti, Anna Pia, Veerle Goossens, Jacques de Mouzon, Siladitya 
//     Bhattacharya, José Antonio Castilla, Vladislav Korsak, Markus Kupka, Karl
//     G. Nygren, Anders Nyboe Andersen, and the European IVF Monitoring
//     Consortium for the European Society of Human Reproduction and Embryology.
//     2012. "Assisted Reproductive Technology in Europe, 2008. Results
//     Generated from European Registers by ESHRE." Human Reproduction 27(9):
//     2571-2584. doi: 10.1093/humrep/des255
// Ferraretti, Anna Pia, Veerle Goossens, Markus Kupka, Siladitya Bhattacharya,
//     Jacques De Mouzon, José Antonio Castilla, Karin Erb, Vladislav Korsak,
//     Anders Nyboe Andersen, and the European IVF Monitoring Consortium for the
//     European Society of Human Reproduction and Embryology. 2013. "Assisted
//     Reproductive Technology in Europe, 2009. Results Generated from European
//     Registers by ESHRE." Human Reproduction 28(9):2318-2331. doi:
//     10.1093/humrep/det278
// Kupka, Markus S., Anna Pia Ferraretti, Jacques De Mouzon, Karin Erb, Thomas
//     D'Hooghe, José Antonio Castilla, Carlos Calhaz-Jorge, Christian De
//     Geyter, Veerle Goossens, and the European IVF Monitoring Consortium for
//     the European Society of Human Reproduction and Embryology. 2014.
//     "Assisted Reproductive Technology in Europe, 2010. Results Generated from
//     European Registers by ESHRE." Human Reproduction 29(10):2099-2113. doi:
//     10.1093/humrep/deu175
// Always the two right-hands columns of Table 1
// "Cycles/million women 15–45 y" and "Cycles/million population"--Cycles of 
// IVF, ICSI, and FER


// Fix country variable
kountry cntry, from(other) stuck marker
ren  _ISO3N_ country
kountry country, from(iso3n) to(iso2c)
ren _ISO2C_ geo
drop country
ren cntry country
ren geo cntry
order cntry country
drop MARKER

// Fix string numbers
replace cycmio1545x2010 = subinstr(cycmio1545x2010, " ", "",.)  // Remove blank (thousands separator)
replace cycmio1545x2009 = subinstr(cycmio1545x2009, " ", "",.)  
replace cycmio1545x2008 = subinstr(cycmio1545x2008, " ", "",.)  
replace cycmio1545x2007 = subinstr(cycmio1545x2007, " ", "",.)  

destring cycmio1545x2010 cycmio1545x2009 cycmio1545x2008 cycmio1545x2007, replace
*list

// Turn data set around
reshape long cycmio1545x cycmio, i(cntry) j(year)
list, sepby(cntry)

// Cross-sectional bar graph at latest point in time
replace cycmio1545x = . if cntry == "DK" & year == 2010 // Get rid of dodgy DK 2010 value

by cntry: egen lastobs = max(cond(!missing(cycmio1545x), year, .))  // Identify year of latest observation
gen byte last = year == lastobs                            // ID variable of latest observation
// According to http://www.stata.com/support/faqs/data-management/first-and-last-occurrences/

graph hbar cycmio1545x if last == 1, over(country, sort(cycmio1545x) descending) nofill ///
      ytitle("ART cycles per million women age 15{c 150}45", size(medsmall)) ylabel(0(2500)15000, ang(45)) ///
      /*note("{it:Note:} Data refer to 2010 or latest available (no earlier than 2007)", span) */ ///
      name(figure1, replace) ysize(6.5) xsize(5.5) scheme(lean1)

// Figure 1: ART cycles per million women age 15–45 per country, 2010
// Sources: Ferraretti et al. (2012; 2013) and Kupka et al. (2014).
// Notes: Values for Albania, Estonia, Ireland, Lithuania, Poland, Spain, 
// Serbia, Switzerland, and Ukraine refer to 2008; for Croatia, Cyprus, France, 
// and Denmark to 2009.  ART cycles refer to IVF, ICSI, frozen embryo 
// replacement (thawings), preimplantation genetic diagnosis and screening, egg
// donation (donation cycles), in vitro maturation, and frozen oocyte
// replacement (thawings).

Oct 2, 2014

Random graphs (31): Stacked bar graph

// Read in data
input str50 mainfield men women
"Engineering and manufacturing" 0.8114392 0.1885608
"Agriculture and veterinary" 0.6366048 0.3633952
"Science, math, and computing" 0.5956416 0.4043584
"Unknown" 0.4658494 0.5341506
"General" 0.4364162 0.5635838
"Service" 0.4277022 0.5722978
"Arts and humanities" 0.3742812 0.6257188
"Social, business, and law" 0.3351495 0.6648505
"Education" 0.2417112 0.7582888
"Health and welfare" 0.1602837 0.8397163
end

// Convert to percentage
replace women = women * 100
replace men = men * 100

// 
graph hbar men women, over(mainfield, sort(women)) ///
                      legend(order(1 "Men" 2 "Women") pos(6) row(1)) ///
                      stack ///
                      ytitle("%") yalternate

Mar 31, 2013

Random graphs (12): Stacked bar graphs

use ahm2009_wide, clear
 
replace country_str = "EL" if country_str == "GR"
 
quietly tab contracttype, gen(contr) 
ren contr1 perma 
ren contr2 tempo 
ren contr3 selfe 
ren contr4 famil
 
collapse perma tempo selfe famil [iw=eu29weight], by(country_str)
 
replace perma = perma * 100 
replace tempo = tempo * 100 
replace selfe = selfe * 100 
replace famil = famil * 100
 
egen order = rank(-perma), unique 
labmask order, value(country_str)
 
graph hbar perma tempo selfe famil ///
   , stack /// Stack bars on top of one another
   over(order, label(labsize(*0.9))) /// // Reduce size of country abbreviations
   ytitle("Percentage first contract") /// 
   legend(label(1 "Permanent") /// 
          label(2 "Temporary") /// 
          label(3 "Self-employed") /// 
          label(4 "Family worker") /// 
          col(2) pos(6)) ///  
   xsize(8.25) ysize(10) /// 
   note("{it:Source:} EU-LFS AHM 2009 (weighted)", span) /// 
   name(contract, replace) 

Mar 18, 2013

Random graphs (9): Bar graphs

clear

// Copy and paste data from here:
// http://epp.eurostat.ec.europa.eu/cache/
// ITY_PUBLIC/3-07032013-AP/EN/3-07032013-AP-EN.PDF

input str30 geo ///
      teachers_primary ///
   teachers_secondary ///
   teachers_tertiary ///
   physicians ///
   managers
EU27 85 59 40 45 33
Belgium 81 61 45 36 30
Bulgaria 94 78 48 . 37
"Czech Republic" 97 58 36 54 26
Denmark 69 . . 45 28
Germany 84 50 39 42 30
Estonia 93 73 . 74 36
Ireland 85 65 . 39 34
Greece . . . 39 23
Spain 75 50 40 51 30
France 83 54 36 41 40
Italy 96 63 36 37 25
Cyprus 83 59 39 . 15
Latvia 93 80 59 74 45
Lithuania 96 79 55 70 39
Luxembourg 74 50 43 30 25
Hungary 96 65 38 54 41
Malta 85 43 30 37 24
Netherlands 85 50 40 45 30
Austria 90 53 38 44 27
Poland 85 66 43 57 39
Portugal 80 68 44 51 34
Romania 86 67 46 69 31
Slovenia 97 66 39 59 38
Slovakia 89 71 44 56 31
Finland 79 58 50 55 32
Sweden 82 52 43 45 35
"United Kingdom" 87 60 43 45 35
Iceland 81 52 47 33 41
Liechtenstein 78 44 27 . .
Norway 74 50 43 43 32
Switzerland 81 40 37 36 33
end

// Label variables
label var geo "Country"
label var teachers_primary "Female teachers in primary education"
label var teachers_secondary "Female teachers in upper secondary education"
label var teachers_tertiary "Female academic staff in tertiary education"
label var physicians "Female physicians (2010)"
label var managers "Female managers"

// Country variable
kountry geo, from(other) stuck
ren _ISO3N_ ctry
kountry ctry, from(iso3n) to(iso2c)
replace _ISO2C_ = "EU-27" if geo == "EU27"
drop ctry
ren _ISO2C_ cntry

drop if cntry == "LI"

// Draw graph
graph hbar managers, over(cntry, sort(1) desc) ///
  xsize(5.5) ysize(8) ///
  caption("Source: Eurostat News Release 37/2013", span) ///
/*note("Includes both corporate managers ///
  and managers of small enterprises", span)*/ ///
  ytitle("Proportion of female managers in 2011")

// Black bar created via the graph editor