pydata

Keep Looking, Don't Settle

jupyter and pandas display

1. show all the rows or columns from a DataFrame in Jupyter QTConcole

if the df has a lot of rows or columns, then when you try to show the df, pandas will auto detect the size of the displaying area and automatically hide some part of the data by replacing with .... To show the full data without any hiding, you can use pd.set_option('display.max_rows', 500) and pd.set_option('display.max_rows', 500) to change the max number of rows or max number of columns to display.

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.random(200).reshape(2, 100))
df

png

# set up display area to show dataframe in jupyter qtconsole

pd.set_option('display.height', 1000)
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

df

png

2. display all text in a cell without truncation

pandas will automatically truncate the long string to display by default. Taking the example below, the string_x is long so by default it will not display the full string. However the full text is wanted. pd.set_option('display.max_colwidth', -1) will help to show all the text strings in the column.

string_x = "if the df has a lot of rows or columns, then when you try to show the df, pandas will auto detect \
the size of the displaying area and automatically hide some part of the data by replacing with"

pd.DataFrame({'string_x': string_x}, index = [0])
string_x
0 if the df has a lot of rows or columns, then w...
pd.options.display.max_rows
pd.set_option('display.max_colwidth', -1)

pd.DataFrame({'string_x': string_x}, index = [0])
string_x
0 if the df has a lot of rows or columns, then when you try to show the df, pandas will auto detect the size of the displaying area and automatically hide some part of the data by replacing with

3. display html in Jupyter

If the dataframe is shown in html, then it is easier to copy the splited data to excel. Otherwise data copied to excel will be in one row. To display in html in jupyter, use from IPython.display import display, HTML

from IPython.display import display, HTML
display(HTML(' <span style="color:red">the title is: <h1>Hello, world!</h1> </span>  '))

the title is:

Hello, world!

4. jupyter run magic command

Jupyter has some magic command making your job easier. Some useful commands like %lsmagic %env %ls %pwd %cd

%lsmagic
Available line magics:
%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

5. run shell command

jupyter also provides the facility to run linux shell command in the way !shall_command.

! ls
20160625_deploy_flask_on_heroku.txt
20170318_pandas_multiple_agg_summary.ipynb
20170402_dl_notes.ipynb
20170402_jupyter_pandas_display.ipynb
build_by_pelican.md
figures

6. list function definiton

?? gives the facility to check the difinicion of a function.

??pd.merge

7. download the server files

FileLink will return a link to the file on your remote server in your browser. You can then click on it to download the file.

from IPython.display import FileLink
FileLink('build_by_pelican.md')

build_by_pelican.md

Reference

  1. Python pandas, how to widen output display to see more columns?

  2. Advanced Jupyter Notebook Tricks — Part I

  3. Advanced Jupyter Notebook Tricks — Part II

  4. Timing and Profiling in IPython