Hi Jody and Ben,
thanks for your answers.
I tried to use pcolormesh instead of pcolor and the result is very good! For
what concern with the memory system problem, I wasn't able to solve it. When I
tried to use the bigger file, I got the same problem. Attached you will find
the script that I'm using to make the plot. May be, I didn't understand very
well how can I use the mmap function.
Regards,
Raffaele.
-----Original Message-----
From: Jody Klymak [mailto:jkly...@uvic.ca]
Sent: Mon 9/8/2014 5:46 PM
To: Benjamin Root
Cc: Raffaele Quarta; Matplotlib Users
Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
It looks like you are calling `pcolor`. Can I suggest you try `pcolormesh`? ii
75 Mb is not a big file!
Cheers, Jody
On Sep 8, 2014, at 7:38 AM, Benjamin Root <ben.r...@ou.edu> wrote:
> (Keeping this on the mailing list so that others can benefit)
>
> What might be happening is that you are keeping around too many numpy arrays
> in memory than you actually need. Take advantage of memmapping, which most
> netcdf tools provide by default. This keeps the data on disk rather than in
> RAM. Second, for very large images, I would suggest either pcolormesh() or
> just simply imshow() instead of pcolor() as they are more way more efficient
> than pcolor(). In addition, it sounds like you are dealing with re-sampled
> data ("at different zoom levels"). Does this mean that you are re-running
> contour on re-sampled data? I am not sure what the benefit of doing that is
> if one could just simply do the contour once at the highest resolution.
>
> Without seeing any code, though, I can only provide generic suggestions.
>
> Cheers!
> Ben Root
>
>
> On Mon, Sep 8, 2014 at 10:12 AM, Raffaele Quarta <raffaele.qua...@linksmt.it>
> wrote:
> Hi Ben,
>
> sorry for the few details that I gave to you. I'm trying to make a contour
> plot of a variable at different zoom levels by using high resolution data.
> The aim is to obtain .PNG output images. Actually, I'm working with big data
> (NetCDF file, dimension is about 75Mb). The current Matplotlib version on my
> UBUNTU 14.04 machine is the 1.3.1 one. My system has a RAM capacity of 8Gb.
> Actually, I'm dealing with memory system problems when I try to make a plot.
> I got the error message as follow:
>
> --------------------------------------------
> cs = m.pcolor(xi,yi,np.squeeze(t))
> File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py", line
> 521, in with_transform
> return plotfunc(self,x,y,data,*args,**kwargs)
> File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py", line
> 3375, in pcolor
> x = ma.masked_values(np.where(x > 1.e20,1.e20,x), 1.e20)
> File "/usr/lib/python2.7/dist-packages/numpy/ma/core.py", line 2195, in
> masked_values
> condition = umath.less_equal(mabs(xnew - value), atol + rtol *
> mabs(value))
> MemoryError
> --------------------------------------------
>
> Otherwise, when I try to make a plot of smaller file (such as 5Mb), it works
> very well. I believe that it's not something of wrong in the script. It might
> be a memory system problem.
> I hope that my message is more clear now.
>
> Thanks for the help.
>
> Regards,
>
> Raffaele
>
> -----------------------------------------
>
> Sent: Mon 9/8/2014 3:19 PM
> To: Raffaele Quarta
> Cc: Matplotlib Users
> Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
>
>
>
> You will need to be more specific... much more specific. What kind of plot
> are you making? How big is your data? What version of matplotlib are you
> using? How much RAM do you have available compared to the amount of data
> (most slowdowns are actually due to swap-thrashing issues). Matplotlib can
> be used for large data, but there exists some speciality tools for the
> truly large datasets. The solution depends on the situation.
>
> Ben Root
>
> On Mon, Sep 8, 2014 at 7:45 AM, Raffaele Quarta <raffaele.qua...@linksmt.it>
> wrote:
>
> > Hi,
> >
> > I'm working with NetCDF format. When I try to make a plot of very large
> > file, I have to wait for a long time for plotting. How can I solve this?
> > Isn't there a solution for this problem?
> >
> > Raffaele
> >
> > --
> > This email was Virus checked by Astaro Security Gateway.
> > http://www.sophos.com
> >
> >
> >
> > ------------------------------------------------------------------------------
> > Want excitement?
> > Manually upgrade your production database.
> > When you want reliability, choose Perforce
> > Perforce version control. Predictably reliable.
> >
> > http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Matplotlib-users mailing list
> > Matplotlib-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> >
> >
>
> --
> This email was Virus checked by Astaro Security Gateway. http://www.sophos.com
>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce
> Perforce version control. Predictably reliable.
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk_______________________________________________
> Matplotlib-users mailing list
> Matplotlib-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
--
Jody Klymak
http://web.uvic.ca/~jklymak/
from netCDF4 import Dataset
import numpy as np
import matplotlib.pyplot as plt
import mmap
#from scipy.io.netcdf import netcdf_file as Dataset
my_example_nc_file = '/home/quartar/Documents/my_exercise/magics/MyOifE140706_140707_0000_03h_01t_T2km.nc'
fh = Dataset(my_example_nc_file, mode='r', mmap=True)
lons = fh.variables['lon'][:]
lats = fh.variables['lat'][:]
t = fh.variables['votemper'][:]
t_units = fh.variables['votemper'].units
fh.close()
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
lon_0 = lons.mean()
lat_0 = lats.mean()
m = Basemap(projection='merc',llcrnrlat=31.95216223802497,urcrnrlat=40.979898069620155,\
llcrnrlon=11.25,urcrnrlon=22.5,lat_ts=20,resolution='h',lat_0=lat_0,lon_0=lon_0)
fig1 = plt.figure(figsize=(1.1025,1.1025))
lon, lat = np.meshgrid(lons, lats)
xi, yi = m(lon, lat)
cs = m.pcolormesh(xi,yi,np.squeeze(t))
plt.axis('off')
plt.savefig('mySeaT', dpi=300, format='png', transparent=True, bbox_inches='tight', pad_inches=0.)
plt.show()
#figure = plt.gcf()
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce.
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users
