Slow performance with plotly chart builder

[helloIAmPau]

Hello all,
I need to use plotly as the backend of a microservice who generates charts dynamically.
Unfortunately, after a little benchmarking, I found that the plotly.express framework is very slow (around 5 secs to generate a chart from 500 lines dataset).

Here is the script I use to generate a scatter matrix:

import sys
import os
import traceback
import json
import time
sys.path.append('c:\\statwolf\\python\packages\Lib\site-packages')
input = json.loads('{\"file\":\"/tmp/data.tsv",\"color\":\"club_country\",\"dimensions\":[\"nolo\",\"tolo\",\"yolo\"]}')
def action():
    def run():
        import plotly.express as px
        from pandas import read_csv
    
        color = None if input['color'] == "" else input['color']
    
        first = time.time()
        d = read_csv(input['file'], sep='\t')
        second = time.time()
        fig = px.scatter_matrix(d, dimensions=input['dimensions'], color=color)
        third = time.time()
        j = fig.to_json()
        fourth = time.time()
    
        print('read: ' + str(second - first))
        print('plot: ' + str(third - second))
        print('json: ' + str(fourth - third))
        return j
    
    import time
    
    for i in range(0, 3):
        start = time.time()
        result = run()
        end = time.time()
    
        print('iteration: ' + str(i) + '\ntime: ' + str(end - start))
    
    return result
    
result = None
try:
    result = { 'outcome': action() }
except Exception as e:
    traceback.print_exc()
    result = { 'error': str(e) }
resultDir = os.path.dirname(os.path.realpath(__file__))
resultFile = open(resultDir + '/result.json', 'w')
json.dump(result, resultFile)
resultFile.close()

from the dataset:
https://www.dropbox.com/s/cm9i3pfv10exbba/data.tsv?dl=1

and this is the report with timing:
https://www.dropbox.com/s/l2x3jqzea4i4xqw/report.txt?dl=1

Now:

1. Is there any tweak I can implement to improve performances?
2. Do you plan to focus on speed for the following releases?

[WenyinWei]

Hello all,
I am also disappointed at the performance of plotly.py because my dataset is not that small. plotly only utilized one of my CPUs to exectute its chart drawing. Does plotly plan to improve the performance by parallel processing or something else?

[ievgennaida]

The same issue.
18 seconds for 800 data points for my data_frame with 10+ columns.
Absolutely unreasonable timings for such small data sets.

        settings["x"] = "time"
        settings["y"] = "value"
        settings["labels"] = {"value": y_title, "time_converted": x_title}
        settings["facet_col"] = "temperature"
        temperatures = str(sorted(data_frame["temperature"].astype(int).unique()))
        plot_settings["category_orders"] = {"temperature": temperatures }

        # information to be appearted in the plot
        settings["hover_data"] = sorted(data_frame.columns)
         px.scatter(data_frame, **settings)

[nicolaskruchten]

I'm sorry that you're seeing such poor performance! In general Plotly Express is much, much faster than this, for example the test suite, which generates hundreds of plots with varying amounts of data, runs in around 30 seconds on my 5-year-old laptop.

@ievgennaida if you can provide a fully-runnable example with data that shows this 18-second performance I'd be happy to take a look.

[ievgennaida]

@nicolaskruchten I cannot share the data but will try to prepare the script as the topic starter did with some fake data.

[ievgennaida]

@nicolaskruchten

I have generated fake data, depends on the PC in our team it's taking 4-8 seconds.
It's taking more than getting a query from the DB.
On a prod, it's slower for me, but might be other reasons.

Basically, I have noticed that using the number group instead of the string is dropping time to 1 second!

        # Fast ->  column_data.append(current_step)
        # Slow -> column_data.append(f"Group Number #{current_step}")

We will also try to use GO scatter with the multithreading to see whether it will help.

import plotly
import time
import plotly.express as px
import json
import random


# Generate fake data:
data_frame = pd.DataFrame()
rows_count = 800
for column in range(10):
    column +=1
    current_step= 0

    data_points_step = 1
    # just generate some groups, not all row should be different
    if column == 2:
        # y
        data_points_step = rows_count / 20
    elif column == 3:
        # color 50 groups
        data_points_step = rows_count / 50
    elif column == 4:
        # facet_col
        data_points_step = rows_count / 5

    column_data = []

    for row in range(rows_count):
        # x
        if column == 1:
            column_data.append(row)
        # y
        elif column == 2:
            column_data.append(current_step)
        # color
        elif column == 3:
            # THIS CAN IMPROVE PERFORMANCE 
            # column_data.append(current_step)
            column_data.append(f"Group Number #{current_step}")
        #  facet_col
        elif column == 4:
            column_data.append(current_step)
        # hover_data
        else:
            column_data.append(f'row-value-{row}')

        if current_step + data_points_step <= row:
            current_step = current_step + data_points_step
    # unsort
    random.shuffle(column_data)
    data_frame[f'x{column}'] = column_data


print(data_frame.shape)

# starting time
start = time.time()


plot_settings = dict()
plot_settings["hover_data"] = sorted(data_frame.columns)
plot_settings["x"] = "x1"
plot_settings["y"] = "x2"
plot_settings["color"] = 'x3'
plot_settings["facet_col"] = "x4"

fig = px.scatter(data_frame, **plot_settings)

figure = json.loads(plotly.io.to_json(fig, validate=True))

# starting time
end = time.time()
print(f"Runtime of the program is {end - start}")
# fig.show()

[jvdd]

Hey @ievgennaida,

I ran your code snippet on my PC, and it finished in 2.866 seconds. The largest chunk (> 90%) of that time is bcs of the px.scatter.

I also tried plotly.express for my use-case (scattering a lot of points; in the following test example there are 2 million points). However, when comparing plotly.express against plotly.grah_objects, I observe that px.scatter is 100x slower than its plotly.graph_objects equivalent.

When further line-profiling px.scatter for this simple use-case (i.e., plotting 1 large sequence), I observed that most of the time is used because of a groupby operation.
As far as I know such operation is not required for creating a scatter plot of 1D array?

If you would wonder why I am interested in this; for plotly-resampler I am looking to create a registration method that adds scalability (under the hood) to the plotly.express interface (see predict-idlab/plotly-resampler#68). However, it seems that plotly.express is significantly slower than plotly its graph_objects interface. I would love to hear some more adivce on how to consume the plotly.express functionality efficiently for very large scatter / line plots?

---

Profiling of px.scatter:

x = np.arange(2_000_000)

%lprun -f px._core.make_figure px.scatter(x)  # line-profile the make_figure function

Profiling result

plotly==5.8.0
python==3.8.10

Timer unit: 1e-06 s

Total time: 18.0257 s
File: /home/jeroen/venv/lib/python3.8/site-packages/plotly/express/_core.py
Function: make_figure at line 1943

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
  1943                                           def make_figure(args, constructor, trace_patch=None, layout_patch=None):
  1944         1         75.0     75.0      0.0      trace_patch = trace_patch or {}
  1945         1          6.0      6.0      0.0      layout_patch = layout_patch or {}
  1946         1       1691.0   1691.0      0.0      apply_default_cascade(args)
  1947                                           
  1948         1    1035154.0 1035154.0      5.7      args = build_dataframe(args, constructor)
  1949         1        602.0    602.0      0.0      if constructor in [go.Treemap, go.Sunburst, go.Icicle] and args["path"] is not None:
  1950                                                   args = process_dataframe_hierarchy(args)
  1951         1          8.0      8.0      0.0      if constructor == "timeline":
  1952                                                   constructor = go.Bar
  1953                                                   args = process_dataframe_timeline(args)
  1954                                           
  1955         2       3157.0   1578.5      0.0      trace_specs, grouped_mappings, sizeref, show_colorbar = infer_config(
  1956         1          8.0      8.0      0.0          args, constructor, trace_patch, layout_patch
  1957                                               )
  1958         1         24.0     24.0      0.0      grouper = [x.grouper or one_group for x in grouped_mappings] or [one_group]
  1959         1    3078711.0 3078711.0     17.1      grouped = args["data_frame"].groupby(grouper, sort=False)
  1960                                           
  1961         1   10608398.0 10608398.0     58.9      orders, sorted_group_names = get_orderings(args, grouper, grouped)
  1962                                           
  1963         1          6.0      6.0      0.0      col_labels = []
  1964         1          3.0      3.0      0.0      row_labels = []
  1965         1          4.0      4.0      0.0      nrows = ncols = 1
  1966         6         26.0      4.3      0.0      for m in grouped_mappings:
  1967         5         23.0      4.6      0.0          if m.grouper not in orders:
  1968         4         22.0      5.5      0.0              m.val_map[""] = m.sequence[0]
  1969                                                   else:
  1970         1          4.0      4.0      0.0              sorted_values = orders[m.grouper]
  1971         1          5.0      5.0      0.0              if m.facet == "col":
  1972                                                           prefix = get_label(args, args["facet_col"]) + "="
  1973                                                           col_labels = [prefix + str(s) for s in sorted_values]
  1974                                                           ncols = len(col_labels)
  1975         1          4.0      4.0      0.0              if m.facet == "row":
  1976                                                           prefix = get_label(args, args["facet_row"]) + "="
  1977                                                           row_labels = [prefix + str(s) for s in sorted_values]
  1978                                                           nrows = len(row_labels)
  1979         2          9.0      4.5      0.0              for val in sorted_values:
  1980         1          5.0      5.0      0.0                  if val not in m.val_map:  # always False if it's an IdentityMap
  1981         1         10.0     10.0      0.0                      m.val_map[val] = m.sequence[len(m.val_map) % len(m.sequence)]
  1982                                           
  1983         1       1749.0   1749.0      0.0      subplot_type = _subplot_type_for_trace_type(constructor().type)
  1984                                           
  1985         1         14.0     14.0      0.0      trace_names_by_frame = {}
  1986         1         13.0     13.0      0.0      frames = OrderedDict()
  1987         1          8.0      8.0      0.0      trendline_rows = []
  1988         1          8.0      8.0      0.0      trace_name_labels = None
  1989         1         14.0     14.0      0.0      facet_col_wrap = args.get("facet_col_wrap", 0)
  1990         2         19.0      9.5      0.0      for group_name in sorted_group_names:
  1991         1    2456203.0 2456203.0     13.6          group = grouped.get_group(group_name if len(group_name) > 1 else group_name[0])
  1992         1         20.0     20.0      0.0          mapping_labels = OrderedDict()
  1993         1          9.0      9.0      0.0          trace_name_labels = OrderedDict()
  1994         1          8.0      8.0      0.0          frame_name = ""
  1995         6         55.0      9.2      0.0          for col, val, m in zip(grouper, group_name, grouped_mappings):
  1996         5         31.0      6.2      0.0              if col != one_group:
  1997         1         45.0     45.0      0.0                  key = get_label(args, col)
  1998         1         25.0     25.0      0.0                  if not isinstance(m.val_map, IdentityMap):
  1999         1         15.0     15.0      0.0                      mapping_labels[key] = str(val)
  2000         1          5.0      5.0      0.0                      if m.show_in_trace_name:
  2001         1          4.0      4.0      0.0                          trace_name_labels[key] = str(val)
  2002         1          4.0      4.0      0.0                  if m.variable == "animation_frame":
  2003                                                               frame_name = val
  2004         1         19.0     19.0      0.0          trace_name = ", ".join(trace_name_labels.values())
  2005         1         10.0     10.0      0.0          if frame_name not in trace_names_by_frame:
  2006         1         11.0     11.0      0.0              trace_names_by_frame[frame_name] = set()
  2007         1          4.0      4.0      0.0          trace_names = trace_names_by_frame[frame_name]
  2008                                           
  2009         2         13.0      6.5      0.0          for trace_spec in trace_specs:
  2010                                                       # Create the trace
  2011         1        653.0    653.0      0.0              trace = trace_spec.constructor(name=trace_name)
  2012         2         23.0     11.5      0.0              if trace_spec.constructor not in [
  2013         1        286.0    286.0      0.0                  go.Parcats,
  2014         1        213.0    213.0      0.0                  go.Parcoords,
  2015         1        167.0    167.0      0.0                  go.Choropleth,
  2016         1         64.0     64.0      0.0                  go.Choroplethmapbox,
  2017         1         70.0     70.0      0.0                  go.Densitymapbox,
  2018         1         92.0     92.0      0.0                  go.Histogram2d,
  2019         1        170.0    170.0      0.0                  go.Sunburst,
  2020         1        142.0    142.0      0.0                  go.Treemap,
  2021         1        147.0    147.0      0.0                  go.Icicle,
  2022                                                       ]:
  2023         2       2045.0   1022.5      0.0                  trace.update(
  2024         1          4.0      4.0      0.0                      legendgroup=trace_name,
  2025         1          7.0      7.0      0.0                      showlegend=(trace_name != "" and trace_name not in trace_names),
  2026                                                           )
  2027         1        789.0    789.0      0.0              if trace_spec.constructor in [go.Bar, go.Violin, go.Box, go.Histogram]:
  2028                                                           trace.update(alignmentgroup=True, offsetgroup=trace_name)
  2029         1          5.0      5.0      0.0              trace_names.add(trace_name)
  2030                                           
  2031                                                       # Init subplot row/col
  2032         1         12.0     12.0      0.0              trace._subplot_row = 1
  2033         1          5.0      5.0      0.0              trace._subplot_col = 1
  2034                                           
  2035         6         57.0      9.5      0.0              for i, m in enumerate(grouped_mappings):
  2036         5         44.0      8.8      0.0                  val = group_name[i]
  2037         5         41.0      8.2      0.0                  try:
  2038         5       5480.0   1096.0      0.0                      m.updater(trace, m.val_map[val])  # covers most cases
  2039                                                           except ValueError:
  2040                                                               # this catches some odd cases like marginals
  2041                                                               if (
  2042                                                                   trace_spec != trace_specs[0]
  2043                                                                   and (
  2044                                                                       trace_spec.constructor in [go.Violin, go.Box]
  2045                                                                       and m.variable in ["symbol", "pattern", "dash"]
  2046                                                                   )
  2047                                                                   or (
  2048                                                                       trace_spec.constructor in [go.Histogram]
  2049                                                                       and m.variable in ["symbol", "dash"]
  2050                                                                   )
  2051                                                               ):
  2052                                                                   pass
  2053                                                               elif (
  2054                                                                   trace_spec != trace_specs[0]
  2055                                                                   and trace_spec.constructor in [go.Histogram]
  2056                                                                   and m.variable == "color"
  2057                                                               ):
  2058                                                                   trace.update(marker=dict(color=m.val_map[val]))
  2059                                                               elif (
  2060                                                                   trace_spec.constructor in [go.Choropleth, go.Choroplethmapbox]
  2061                                                                   and m.variable == "color"
  2062                                                               ):
  2063                                                                   trace.update(
  2064                                                                       z=[1] * len(group),
  2065                                                                       colorscale=[m.val_map[val]] * 2,
  2066                                                                       showscale=False,
  2067                                                                       showlegend=True,
  2068                                                                   )
  2069                                                               else:
  2070                                                                   raise
  2071                                           
  2072                                                           # Find row for trace, handling facet_row and marginal_x
  2073         5         79.0     15.8      0.0                  if m.facet == "row":
  2074         1          7.0      7.0      0.0                      row = m.val_map[val]
  2075                                                           else:
  2076         4         74.0     18.5      0.0                      if (
  2077         4         54.0     13.5      0.0                          args.get("marginal_x") is not None  # there is a marginal
  2078                                                                   and trace_spec.marginal != "x"  # and we're not it
  2079                                                               ):
  2080                                                                   row = 2
  2081                                                               else:
  2082         4         39.0      9.8      0.0                          row = 1
  2083                                           
  2084                                                           # Find col for trace, handling facet_col and marginal_y
  2085         5         84.0     16.8      0.0                  if m.facet == "col":
  2086         1          9.0      9.0      0.0                      col = m.val_map[val]
  2087         1          9.0      9.0      0.0                      if facet_col_wrap:  # assumes no facet_row, no marginals
  2088                                                                   row = 1 + ((col - 1) // facet_col_wrap)
  2089                                                                   col = 1 + ((col - 1) % facet_col_wrap)
  2090                                                           else:
  2091         4         55.0     13.8      0.0                      if trace_spec.marginal == "y":
  2092                                                                   col = 2
  2093                                                               else:
  2094         4         55.0     13.8      0.0                          col = 1
  2095                                           
  2096         5         69.0     13.8      0.0                  if row > 1:
  2097                                                               trace._subplot_row = row
  2098                                           
  2099         5         60.0     12.0      0.0                  if col > 1:
  2100                                                               trace._subplot_col = col
  2101         1         13.0     13.0      0.0              if (
  2102         1        291.0    291.0      0.0                  trace_specs[0].constructor == go.Histogram2dContour
  2103                                                           and trace_spec.constructor == go.Box
  2104                                                           and trace.line.color
  2105                                                       ):
  2106                                                           trace.update(marker=dict(color=trace.line.color))
  2107                                           
  2108         1         11.0     11.0      0.0              if "ecdfmode" in args:
  2109                                                           base = args["x"] if args["orientation"] == "v" else args["y"]
  2110                                                           var = args["x"] if args["orientation"] == "h" else args["y"]
  2111                                                           ascending = args.get("ecdfmode", "standard") != "reversed"
  2112                                                           group = group.sort_values(by=base, ascending=ascending)
  2113                                                           group_sum = group[var].sum()  # compute here before next line mutates
  2114                                                           group[var] = group[var].cumsum()
  2115                                                           if not ascending:
  2116                                                               group = group.sort_values(by=base, ascending=True)
  2117                                           
  2118                                                           if args.get("ecdfmode", "standard") == "complementary":
  2119                                                               group[var] = group_sum - group[var]
  2120                                           
  2121                                                           if args["ecdfnorm"] == "probability":
  2122                                                               group[var] = group[var] / group_sum
  2123                                                           elif args["ecdfnorm"] == "percent":
  2124                                                               group[var] = 100.0 * group[var] / group_sum
  2125                                           
  2126         2      10811.0   5405.5      0.1              patch, fit_results = make_trace_kwargs(
  2127         1         13.0     13.0      0.0                  args, trace_spec, group, mapping_labels.copy(), sizeref
  2128                                                       )
  2129         1      35173.0  35173.0      0.2              trace.update(patch)
  2130         1         10.0     10.0      0.0              if fit_results is not None:
  2131                                                           trendline_rows.append(mapping_labels.copy())
  2132                                                           trendline_rows[-1]["px_fit_results"] = fit_results
  2133         1          9.0      9.0      0.0              if frame_name not in frames:
  2134         1         24.0     24.0      0.0                  frames[frame_name] = dict(data=[], name=frame_name)
  2135         1         13.0     13.0      0.0              frames[frame_name]["data"].append(trace)
  2136         1         24.0     24.0      0.0      frame_list = [f for f in frames.values()]
  2137         1         11.0     11.0      0.0      if len(frame_list) > 1:
  2138                                                   frame_list = sorted(
  2139                                                       frame_list, key=lambda f: orders[args["animation_frame"]].index(f["name"])
  2140                                                   )
  2141                                           
  2142         1          7.0      7.0      0.0      if show_colorbar:
  2143                                                   colorvar = "z" if constructor in [go.Histogram2d, go.Densitymapbox] else "color"
  2144                                                   range_color = args["range_color"] or [None, None]
  2145                                           
  2146                                                   colorscale_validator = ColorscaleValidator("colorscale", "make_figure")
  2147                                                   layout_patch["coloraxis1"] = dict(
  2148                                                       colorscale=colorscale_validator.validate_coerce(
  2149                                                           args["color_continuous_scale"]
  2150                                                       ),
  2151                                                       cmid=args["color_continuous_midpoint"],
  2152                                                       cmin=range_color[0],
  2153                                                       cmax=range_color[1],
  2154                                                       colorbar=dict(
  2155                                                           title_text=get_decorated_label(args, args[colorvar], colorvar)
  2156                                                       ),
  2157                                                   )
  2158         3         25.0      8.3      0.0      for v in ["height", "width"]:
  2159         2         17.0      8.5      0.0          if args[v]:
  2160                                                       layout_patch[v] = args[v]
  2161         1         10.0     10.0      0.0      layout_patch["legend"] = dict(tracegroupgap=0)
  2162         1          7.0      7.0      0.0      if trace_name_labels:
  2163         1         22.0     22.0      0.0          layout_patch["legend"]["title_text"] = ", ".join(trace_name_labels)
  2164         1         43.0     43.0      0.0      if args["title"]:
  2165                                                   layout_patch["title_text"] = args["title"]
  2166         1        617.0    617.0      0.0      elif args["template"].layout.margin.t is None:
  2167         1         12.0     12.0      0.0          layout_patch["margin"] = {"t": 60}
  2168         2         14.0      7.0      0.0      if (
  2169         1          7.0      7.0      0.0          "size" in args
  2170         1          8.0      8.0      0.0          and args["size"]
  2171                                                   and args["template"].layout.legend.itemsizing is None
  2172                                               ):
  2173                                                   layout_patch["legend"]["itemsizing"] = "constant"
  2174                                           
  2175         1         10.0     10.0      0.0      if facet_col_wrap:
  2176                                                   nrows = math.ceil(ncols / facet_col_wrap)
  2177                                                   ncols = min(ncols, facet_col_wrap)
  2178                                           
  2179         1         11.0     11.0      0.0      if args.get("marginal_x") is not None:
  2180                                                   nrows += 1
  2181                                           
  2182         1          8.0      8.0      0.0      if args.get("marginal_y") is not None:
  2183                                                   ncols += 1
  2184                                           
  2185         2      76233.0  38116.5      0.4      fig = init_figure(
  2186         1          8.0      8.0      0.0          args, subplot_type, frame_list, nrows, ncols, col_labels, row_labels
  2187                                               )
  2188                                           
  2189                                               # Position traces in subplots
  2190         2         18.0      9.0      0.0      for frame in frame_list:
  2191         2         14.0      7.0      0.0          for trace in frame["data"]:
  2192         1        338.0    338.0      0.0              if isinstance(trace, go.Splom):
  2193                                                           # Special case that is not compatible with make_subplots
  2194                                                           continue
  2195                                           
  2196         2       2027.0   1013.5      0.0              _set_trace_grid_reference(
  2197         1         11.0     11.0      0.0                  trace,
  2198         1         79.0     79.0      0.0                  fig.layout,
  2199         1         13.0     13.0      0.0                  fig._grid_ref,
  2200         1         16.0     16.0      0.0                  nrows - trace._subplot_row + 1,
  2201         1         11.0     11.0      0.0                  trace._subplot_col,
  2202                                                       )
  2203                                           
  2204                                               # Add traces, layout and frames to figure
  2205         1     493599.0 493599.0      2.7      fig.add_traces(frame_list[0]["data"] if len(frame_list) > 0 else [])
  2206         1      18070.0  18070.0      0.1      fig.update_layout(layout_patch)
  2207         1         24.0     24.0      0.0      if "template" in args and args["template"] is not None:
  2208         1     167201.0 167201.0      0.9          fig.update_layout(template=args["template"], overwrite=True)
  2209         1        150.0    150.0      0.0      fig.frames = frame_list if len(frames) > 1 else []
  2210                                           
  2211         1         11.0     11.0      0.0      if args.get("trendline") and args.get("trendline_scope", "trace") == "overall":
  2212                                                   trendline_spec = make_trendline_spec(args, constructor)
  2213                                                   trendline_trace = trendline_spec.constructor(
  2214                                                       name="Overall Trendline", legendgroup="Overall Trendline", showlegend=False
  2215                                                   )
  2216                                                   if "line" not in trendline_spec.trace_patch:  # no color override
  2217                                                       for m in grouped_mappings:
  2218                                                           if m.variable == "color":
  2219                                                               next_color = m.sequence[len(m.val_map) % len(m.sequence)]
  2220                                                               trendline_spec.trace_patch["line"] = dict(color=next_color)
  2221                                                   patch, fit_results = make_trace_kwargs(
  2222                                                       args, trendline_spec, args["data_frame"], {}, sizeref
  2223                                                   )
  2224                                                   trendline_trace.update(patch)
  2225                                                   fig.add_trace(
  2226                                                       trendline_trace, row="all", col="all", exclude_empty_subplots=True
  2227                                                   )
  2228                                                   fig.update_traces(selector=-1, showlegend=True)
  2229                                                   if fit_results is not None:
  2230                                                       trendline_rows.append(dict(px_fit_results=fit_results))
  2231                                           
  2232         1       2555.0   2555.0      0.0      fig._px_trendlines = pd.DataFrame(trendline_rows)
  2233                                           
  2234         1      20715.0  20715.0      0.1      configure_axes(args, constructor, fig, orders)
  2235         1         14.0     14.0      0.0      configure_animation_controls(args, constructor, fig)
  2236         1          3.0      3.0      0.0      return fig

[dss010101]

this think is horrendously slow w/ financial data of more than few years