BUG#16857395 - EXCESSIVE MEMORY USAGE AFTER REPEATED TRIGGER

               EXCEPTION HANDLERS

Analysis
--------
Excessive memory consumption is observed when executing a
stored procedure multiple times when:

a) The stored procedure has an SQL statement which fails during
   validation.

b) The stored procedure has an an SQL statement which requires to
   be re-prepared.

Under the above scenarios, the query within the stored procedure is
invalidated during the execution. The invalidation calls for the
re-parse of the query. During the re-parse, the lex tree is allocated
on the persistent mem_root of the stored procedure. When the stored
procedure is executed multiple times, the repeated allocation of the
new lex tree on the persistent mem_root is observed. The previous
allocations for the lex tree in the mem_root are never freed. In
addition to the lex tree, the Ref_ptr_array(array of pointers to
the field items) was also allocated on the persistent mem_root and
never freed during re-parse. Hence the memory growth was observed.

Also the patch solves the memory growth due to the cases listed below:
a) Allocation of Sql_condition and its memory on persistent mem-root
   every time a condition is raised.

b) Allocation of Handler-structure on the persistent mem-root every
   time an SQL-handler is pushed. Observed when the execution flow
   goes to DECLARE HANDLER statement.

c) Allocation of Cursor-structure on the persistent mem-root every
   time a cursor is pushed. Observed when the DECLARE CURSOR
   statement is executed.

Fix:
----
1) Patch for the allocation of the lex tree and 'Ref_ptr_array' on the
   persistent mem_root:
   A new mem_root 'm_lex_mem_root' is introduced which holds the lex
   tree and the Ref_ptr_array. When a query is invalidated, before the
   reparse we free this mem_root, init the root and use it to allocate
   the newly parsed lex tree.

   The 'ref_pointer_array' part of the lex tree object points the
   Ref_ptr_array. Since 'Ref_ptr_array' should not be
   allocated on the persistent mem_root, we switch the stmt_arena's
   mem_root to that of the new 'm_lex_mem_root'. This mem_root
   is persistent until reparse or the stored routine is dropped,
   hence the 'Ref_ptr_array' is allocated on the new mem_root as well.
   Since this mem_root is freed and re-initialised during
   re-parse we will not observe a memory growth. Also ensures that
   the prepared statement execution is unaffected.

2) Patch for the next set of issues listed above:
  a) The Sql_condition is a temporary instance hence initialised in
     the execution mem_root which is freed at the end of the execution.
  b) The handler is now stored on the heap instead of the caller's
     mem_root thus avoiding the memory growth.
  c) The cursor is now stored on the heap instead of the caller's
     mem_root thus avoiding the memory growth.

Note: I have not added an mtr test case: A test case which executes
      for a longer period is needed to observe the increasing memory
      consumption and is not suitable for mtr. Also the existing test
      suite has tests which uses the new mem_root.

Author: Nisha Gopalakrishnan

sql/sp_instr.cc

@@ -1,4 +1,4 @@
-/* Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2012, 2013, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -458,6 +458,25 @@ LEX *sp_lex_instr::parse_expr(THD *thd, sp_head *sp)
     return NULL;
   }
 
+  // Cleanup current THD from previously held objects before new parsing.
+  cleanup_before_parsing(thd);
+
+  // Cleanup and re-init the lex mem_root for re-parse.
+  free_root(&m_lex_mem_root, MYF(0));
+  init_sql_alloc(&m_lex_mem_root, MEM_ROOT_BLOCK_SIZE, MEM_ROOT_PREALLOC);
+
+  /*
+    Switch mem-roots. We store the new LEX and its Items in the
+    m_lex_mem_root since it is freed before reparse triggered due to
+    invalidation. This avoids the memory leak in case re-parse is
+    initiated. Also set the statement query arena to the lex mem_root.
+  */
+  MEM_ROOT *execution_mem_root= thd->mem_root;
+  Query_arena parse_arena(&m_lex_mem_root, thd->stmt_arena->state);
+
+  thd->mem_root= &m_lex_mem_root;
+  thd->stmt_arena->set_query_arena(&parse_arena);
+
   // Prepare parser state. It can be done just before parse_sql(), do it here
   // only to simplify exit in case of failure (out-of-memory error).
 
@@ -466,17 +485,6 @@ LEX *sp_lex_instr::parse_expr(THD *thd, sp_head *sp)
   if (parser_state.init(thd, sql_query.c_ptr(), sql_query.length()))
     return NULL;
 
-  // Cleanup current THD from previously held objects before new parsing.
-
-  cleanup_before_parsing(thd);
-
-  // Switch mem-roots. We need to store new LEX and its Items in the persistent
-  // SP-memory (memory which is not freed between executions).
-
-  MEM_ROOT *execution_mem_root= thd->mem_root;
-
-  thd->mem_root= thd->sp_runtime_ctx->sp->get_persistent_mem_root();
-
   // Switch THD::free_list. It's used to remember the newly created set of Items
   // during parsing. We should clean those items after each execution.
 

sql/sp_instr.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2012, 2013, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -177,10 +177,21 @@ class sp_lex_instr : public sp_instr
     m_lex_query_tables_own_last(NULL)
   {
     set_lex(lex, is_lex_owner);
+    memset(&m_lex_mem_root, 0, sizeof (MEM_ROOT));
   }
 
   virtual ~sp_lex_instr()
-  { free_lex(); }
+  {
+    free_lex();
+    /*
+      If the instruction is reparsed, m_lex_mem_root was used to allocate the
+      items, then freeing the memroot, frees the items. Hence set the free_list
+      pointer to NULL.
+    */
+    if (alloc_root_inited(&m_lex_mem_root))
+      free_list= NULL;
+    free_root(&m_lex_mem_root, MYF(0));
+  }
 
   /**
     Make a few attempts to execute the instruction.
@@ -334,6 +345,13 @@ class sp_lex_instr : public sp_instr
   virtual void cleanup_before_parsing(THD *thd);
 
 private:
+  /** 
+    Mem-root for storing the LEX-tree during reparse. This
+    mem-root is freed when a reparse is triggered or the stored
+    routine is dropped.
+  */
+  MEM_ROOT m_lex_mem_root;
+
   /// LEX-object.
   LEX *m_lex;
 

sql/sp_rcontext.cc

@@ -1,4 +1,4 @@
-/* Copyright (c) 2002, 2012, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2002, 2013, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -23,6 +23,7 @@
 #include "sql_tmp_table.h"                     // create_virtual_tmp_table
 #include "sp_instr.h"
 
+extern "C" void sql_alloc_error_handler(void);
 
 ///////////////////////////////////////////////////////////////////////////
 // sp_rcontext implementation.
@@ -51,8 +52,12 @@ sp_rcontext::~sp_rcontext()
   while (m_activated_handlers.elements())
     delete m_activated_handlers.pop();
 
-  // Leave m_visible_handlers, m_var_items, m_cstack
-  // and m_case_expr_holders untouched.
+  while (m_visible_handlers.elements())
+    delete m_visible_handlers.pop();
+   
+  pop_all_cursors();
+
+  // Leave m_var_items and m_case_expr_holders untouched.
   // They are allocated in mem roots and will be freed accordingly.
 }
 
@@ -158,13 +163,19 @@ bool sp_rcontext::set_return_value(THD *thd, Item **return_value_item)
 bool sp_rcontext::push_cursor(sp_instr_cpush *i)
 {
   /*
-    We should create cursors in the callers arena, as
-    it could be (and usually is) used in several instructions.
+    We should create cursors on the system heap because:
+     - they could be (and usually are) used in several instructions,
+       thus they can not be stored on an execution mem-root;
+     - a cursor can be pushed/popped many times in a loop, having these objects
+       on callers' mem-root would lead to a memory leak in every iteration.
   */
-  sp_cursor *c= new (callers_arena->mem_root) sp_cursor(i);
+  sp_cursor *c= new (std::nothrow) sp_cursor(i);
 
-  if (c == NULL)
+  if (!c)
+  {
+    sql_alloc_error_handler();
     return true;
+  }
 
   m_cstack[m_ccount++]= c;
   return false;
@@ -183,14 +194,22 @@ void sp_rcontext::pop_cursors(uint count)
 bool sp_rcontext::push_handler(sp_handler *handler, uint first_ip)
 {
   /*
-    We should create handler entries in the callers arena, as
-    they could be (and usually are) used in several instructions.
+    We should create handler entries on the system heap because:
+      - they could be (and usually are) used in several instructions,
+        thus they can not be stored on an execution mem-root;
+      - a handler can be pushed/popped many times in a loop, having these
+        objects on callers' mem-root would lead to a memory leak in every
+        iteration.
   */
+
   sp_handler_entry *he=
-    new (callers_arena->mem_root) sp_handler_entry(handler, first_ip);
+    new (std::nothrow) sp_handler_entry(handler, first_ip);
 
-  if (he == NULL)
+  if (!he)
+  {
+    sql_alloc_error_handler();
     return true;
+  }
 
   return m_visible_handlers.append(he);
 }
@@ -203,7 +222,7 @@ void sp_rcontext::pop_handlers(sp_pcontext *current_scope)
     int handler_level= m_visible_handlers.at(i)->handler->scope->get_level();
 
     if (handler_level >= current_scope->get_level())
-      m_visible_handlers.pop();
+      delete m_visible_handlers.pop();
   }
 }
 
@@ -249,7 +268,11 @@ bool sp_rcontext::handle_sql_condition(THD *thd,
 
   Diagnostics_area *da= thd->get_stmt_da();
   const sp_handler *found_handler= NULL;
-  const Sql_condition *found_condition= NULL;
+
+  uint condition_sql_errno= 0;
+  Sql_condition::enum_warning_level condition_level;
+  const char *condition_sqlstate;
+  const char *condition_message;
 
   if (thd->is_error())
   {
@@ -259,24 +282,29 @@ bool sp_rcontext::handle_sql_condition(THD *thd,
                                           da->sql_errno(),
                                           Sql_condition::WARN_LEVEL_ERROR);
 
-    if (found_handler)
-      found_condition= da->get_error_condition();
+    if (!found_handler)
+      DBUG_RETURN(false);
 
-    /*
-      Found condition can be NULL if the diagnostics area was full
-      when the error was raised. It can also be NULL if
-      Diagnostics_area::set_error_status(uint sql_error) was used.
-      In these cases, make a temporary Sql_condition here so the
-      error can be handled.
-    */
-    if (!found_condition)
+    if (da->get_error_condition())
     {
-      Sql_condition *condition=
-        new (callers_arena->mem_root) Sql_condition(callers_arena->mem_root);
-      condition->set(da->sql_errno(), da->get_sqlstate(),
-                     Sql_condition::WARN_LEVEL_ERROR,
-                     da->message());
-      found_condition= condition;
+      const Sql_condition *c= da->get_error_condition();
+      condition_sql_errno= c->get_sql_errno();
+      condition_level= c->get_level();
+      condition_sqlstate= c->get_sqlstate();
+      condition_message= c->get_message_text();
+    }
+    else
+    {
+      /*
+        SQL condition can be NULL if the diagnostics area was full
+        when the error was raised. It can also be NULL if
+        Diagnostics_area::set_error_status(uint sql_error) was used.
+      */
+
+      condition_sql_errno= da->sql_errno();
+      condition_level= Sql_condition::WARN_LEVEL_ERROR;
+      condition_sqlstate= da->get_sqlstate();
+      condition_message= da->message();
     }
   }
   else if (da->current_statement_warn_count())
@@ -301,7 +329,11 @@ bool sp_rcontext::handle_sql_condition(THD *thd,
         if (handler)
         {
           found_handler= handler;
-          found_condition= c;
+
+          condition_sql_errno= c->get_sql_errno();
+          condition_level= c->get_level();
+          condition_sqlstate= c->get_sqlstate();
+          condition_message= c->get_message_text();
         }
       }
     }
@@ -314,7 +346,7 @@ bool sp_rcontext::handle_sql_condition(THD *thd,
   //  - there is a pending SQL-condition (error or warning);
   //  - there is an SQL-handler for it.
 
-  DBUG_ASSERT(found_condition);
+  DBUG_ASSERT(condition_sql_errno != 0);
 
   sp_handler_entry *handler_entry= NULL;
   for (int i= 0; i < m_visible_handlers.elements(); ++i)
@@ -363,9 +395,20 @@ bool sp_rcontext::handle_sql_condition(THD *thd,
                                 // (e.g. "bad data").
 
   /* Add a frame to handler-call-stack. */
-  Handler_call_frame *frame= new Handler_call_frame(found_handler,
-                                                    found_condition,
-                                                    continue_ip);
+  Handler_call_frame *frame=
+    new (std::nothrow) Handler_call_frame(found_handler,
+                                          condition_sql_errno,
+                                          condition_sqlstate,
+                                          condition_level,
+                                          condition_message,
+                                          continue_ip);
+
+  if (!frame)
+  {
+    sql_alloc_error_handler();
+    DBUG_RETURN(false);
+  }
+
   m_activated_handlers.append(frame);
 
   *ip= handler_entry->first_ip;

sql/sp_rcontext.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2002, 2012, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2002, 2013, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -79,7 +79,7 @@ class sp_rcontext : public Sql_alloc
 private:
   /// This is an auxillary class to store entering instruction pointer for an
   /// SQL-handler.
-  class sp_handler_entry : public Sql_alloc
+  class sp_handler_entry
   {
   public:
     /// Handler definition (from parsing context).
@@ -130,16 +130,21 @@ class sp_rcontext : public Sql_alloc
 
     /// The constructor.
     ///
-    /// @param _sql_condition  The SQL condition.
-    /// @param arena           Query arena for SP
-    Sql_condition_info(const Sql_condition *_sql_condition)
-      :sql_errno(_sql_condition->get_sql_errno()),
-       level(_sql_condition->get_level())
+    /// @param _sql_errno  SQL error number.
+    /// @param _sql_state  Diagnostic status.
+    /// @param _level      Error level of the condition.
+    /// @param _message    Error message.
+    Sql_condition_info(uint _sql_errno,
+                       const char *_sql_state,
+                       Sql_condition::enum_warning_level _level,
+                       const char *_message)
+     :sql_errno(_sql_errno),
+      level(_level)
     {
-      memcpy(sql_state, _sql_condition->get_sqlstate(), SQLSTATE_LENGTH);
+      memcpy(sql_state, _sql_state, SQLSTATE_LENGTH);
       sql_state[SQLSTATE_LENGTH]= '\0';
 
-      strncpy(message, _sql_condition->get_message_text(), MYSQL_ERRMSG_SIZE);
+      strncpy(message, _message, MYSQL_ERRMSG_SIZE);
     }
   };
 
@@ -154,21 +159,28 @@ class sp_rcontext : public Sql_alloc
     const sp_handler *handler;
 
     /// SQL-condition, triggered handler activation.
-    const Sql_condition_info sql_condition;
+    Sql_condition_info sql_condition_info;
 
     /// Continue-instruction-pointer for CONTINUE-handlers.
     /// The attribute contains 0 for EXIT-handlers.
     uint continue_ip;
 
     /// The constructor.
     ///
-    /// @param _sql_condition SQL-condition, triggered handler activation.
-    /// @param _continue_ip   Continue instruction pointer.
+    /// @param _handler      handler triggered due to SQL-condition.
+    /// @param _sql_errno    SQL error number.
+    /// @param _sql_state    Diagnostic status.
+    /// @param  _level       Error level of the condition.
+    /// @param  _message     Error message.
+    /// @param _continue_ip  Continue instruction pointer.
     Handler_call_frame(const sp_handler *_handler,
-                       const Sql_condition *_sql_condition,
+                       uint _sql_errno,
+                       const char *_sql_state,
+                       Sql_condition::enum_warning_level _level,
+                       const char *_message,
                        uint _continue_ip)
      :handler(_handler),
-      sql_condition(_sql_condition),
+      sql_condition_info(_sql_errno, _sql_state, _level, _message),
       continue_ip(_continue_ip)
     { }
  };
@@ -229,7 +241,7 @@ class sp_rcontext : public Sql_alloc
   const Sql_condition_info *raised_condition() const
   {
     return m_activated_handlers.elements() ?
-      &(*m_activated_handlers.back())->sql_condition : NULL;
+      &(*m_activated_handlers.back())->sql_condition_info : NULL;
   }
 
   /// Handle current SQL condition (if any).
@@ -420,7 +432,7 @@ typedef class st_select_lex_unit SELECT_LEX_UNIT;
 
 /* A mediator between stored procedures and server side cursors */
 
-class sp_cursor : public Sql_alloc
+class sp_cursor
 {
 private:
   /// An interceptor of cursor result set used to implement
@@ -468,6 +480,6 @@ class sp_cursor : public Sql_alloc
 
 private:
   void destroy();
-}; // class sp_cursor : public Sql_alloc
+}; // class sp_cursor
 
 #endif /* _SP_RCONTEXT_H_ */